AR-based muscle relaxation state visual guidance system

Abstract

The embodiment of the invention provides a muscle relaxation state visual guiding system based on AR, and relates to the technical field of medical auxiliary device technologies. The system comprises a muscle relaxation monitoring module, an image sensing module, an image processing module and an augmented reality display terminal. According to the invention, the problems of high sensing delay and low coordination efficiency are solved, and the effects of reducing the sensing delay and improving the coordination efficiency are achieved.

Description

Technical Field

[0001] The present invention relates to the field of medical assistive devices, and more specifically, to an AR-based visualization and guidance system for muscle relaxation. Background Technology

[0002] In modern surgical anesthesia management, ensuring the patient is in an appropriate state of muscle relaxation is crucial for the smooth progress of the operation. Excessive muscle relaxation may lead to delayed postoperative recovery or complications, while insufficient muscle relaxation may cause patient movement during surgery, severely interfering with the procedure and increasing patient risk. Traditional muscle relaxation monitoring relies on physical monitors, which electrically stimulate the patient's nerves and measure muscle contraction responses, outputting numerical indicators such as the "Train-of-Four (TOF) ratio." Anesthesiologists interpret these abstract numbers, combining them with clinical experience, to determine the depth of muscle relaxation and adjust drug dosages.

[0003] However, existing monitoring models have significant flaws.

[0004] First, during surgery, anesthesiologists must simultaneously monitor changes in the patient's vital signs within the surgical field and the digital indicators on the physical monitor screen located next to the operating table. This frequent switching of visual focus not only increases the surgeon's cognitive load, but each shift in gaze is accompanied by head movement, eye refocusing, and cognitive context reconstruction, resulting in a perceptual interruption of approximately 300ms to 500ms. In critical surgical steps, this delay can lead to a lag in responding to changes in the patient's muscle relaxation depth, impacting surgical safety.

[0005] Secondly, there is an asynchrony in the acquisition of information about muscle relaxation status between surgeons and anesthesiologists. Surgeons primarily focus on the surgical site, while anesthesiologists focus on monitor data. The lack of a unified and intuitive spatial reference limits the efficiency and real-time nature of collaborative decision-making. When insufficient muscle relaxation leads to movement risks, verbal delays in information transmission and the uncertainty of spatial location limit decision-making efficiency. Furthermore, the original "four-stranded stimulus ratio" is merely an abstract number, lacking an intuitive physiological spatial association. Doctors need to perform secondary mapping in their minds, increasing their cognitive burden.

[0006] Therefore, designing a technical solution to achieve intuitive, immersive, and patient-space-anchored monitoring of muscle relaxation, while resolving friction points in medical-nursing information collaboration, has become an urgent technical solution in the field of clinical anesthesia, in order to reduce medical risks and optimize surgical procedures. Summary of the Invention

[0007] This invention provides an AR-based visualization guidance system for muscle relaxation status, which at least solves the problem in related technologies where muscle relaxation status monitoring information is separated from the surgical field of vision, resulting in perception delay and low coordination efficiency.

[0008] According to one embodiment of the present invention, an AR-based visualization and guidance system for muscle relaxation is provided, comprising: The muscle relaxation monitoring module is used to collect patients' muscle relaxation monitoring data in real time and output four sets of stimulation ratios; The image perception module is used to capture real-time environmental images of the surgical field and acquire pose tracking data; An image processing module, connected to the muscle relaxation monitoring module and the image perception module, is used to construct a real-time surgical space coordinate system, convert the four series of stimulus ratios into a virtual information layer, and calculate the projection parameters of the virtual information layer; An augmented reality display terminal, connected to the image processing module, is used to overlay the virtual information layer onto the physical target location within the target's field of view.

[0009] In one exemplary embodiment, the image processing module includes: The data preprocessing unit is used to preprocess the ratios of the four cascaded stimuli. Spatial anchoring unit is used to calculate the coordinate mapping relationship between virtual objects and the physical world based on image information and pose data; The rendering engine unit is used to generate virtual instruments or color maps with dynamic visual effects in real time based on the preprocessed values ​​of the ratios of four strings of stimuli.

[0010] In one exemplary embodiment, the augmented reality display terminal employs an optical waveguide optics component to guide the microdisplay signal generated by the image processing module to the retina of the human eye while maintaining the transmittance of external ambient light.

[0011] In one exemplary embodiment, the image processing module further includes: A visual data integrity monitoring unit is used to: receive multimodal physiological data related to the patient's physiological state, wherein the multimodal physiological data includes at least one of heart rate, blood oxygen saturation, or end-tidal carbon dioxide partial pressure; An evaluation unit is used to evaluate the consistency between the muscle relaxation state displayed by the virtual information layer and the multimodal physiological data, so as to generate a physiological consistency risk score. The monitoring unit is used to continuously monitor the visual spatial position stability of the virtual information layer relative to the physical anchoring target, so as to generate a spatial stability risk score. The overlay layer unit is used to control the augmented reality display terminal to generate and overlay an abnormal warning layer when the physiological consistency risk score or spatial stability risk score exceeds a preset threshold.

[0012] According to another embodiment of the present invention, an AR-based visualization guidance system method for muscle relaxation state is provided, comprising: Acquire real-time muscle relaxation monitoring data of patients collected by a muscle relaxation monitoring device, wherein the real-time muscle relaxation monitoring data includes four series of stimulation ratios; Capture image information of the surgical field of view, and construct a real-time surgical space coordinate system containing physically anchored targets based on the image information; Four sets of stimulus ratios are input into a preset visual mapping model to generate a corresponding virtual information layer, wherein the virtual information layer includes color features and graphic geometric features associated with the values ​​of the four sets of stimulus ratios. Calculate the projection parameters of the virtual information layer in the augmented reality glasses display unit based on the real-time surgical space coordinate system; The augmented reality glasses are used to overlay and project the virtual information layer onto a preset position in the surgical field of view to achieve immersive monitoring of the patient's muscle relaxation state.

[0013] In one exemplary embodiment, constructing a real-time surgical space coordinate system containing physically anchored targets based on the image information includes: Multi-frame video streams of the surgical field of view are obtained through the external camera of augmented reality glasses; Identify preset physical anchoring markers in the video stream, or identify surface features of the patient's surgical site; Establish the rotation and translation transformation matrix between the world coordinate system of the augmented reality glasses and the physical anchor target.

[0014] In an exemplary embodiment, the step of inputting the four sequential stimulus ratios into a preset visual mapping model to generate a corresponding virtual information layer includes: The four cascaded stimulus ratios are mapped to a preset continuous color space; The pointer deflection angle of the virtual dashboard is determined based on the ratio of the four sets of stimuli, or the transparency parameter or flashing frequency of the virtual color block is determined.

[0015] In one exemplary embodiment, the method further includes: Acquire multimodal physiological data related to the patient's physiological state, wherein the multimodal physiological data includes at least one of heart rate, blood oxygen saturation, or end-tidal carbon dioxide partial pressure; The consistency between the muscle relaxation state displayed by the virtual information layer and the multimodal physiological data is evaluated to generate a physiological consistency risk score; The visual spatial position stability of the virtual information layer relative to the physical anchored target is continuously monitored to generate a spatial stability risk score. If the physiological consistency risk score or spatial stability risk score exceeds a preset threshold, an abnormality warning layer is generated and overlaid in the augmented reality glasses.

[0016] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0017] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0018] This invention utilizes AR technology to transform abstract muscle relaxation monitoring indicators into intuitive virtual visual layers bound to the patient's body space, eliminating friction points caused by doctors switching gaze. By mapping four sets of stimulus ratios in real time to colors or dashboards, doctors can obtain the patient's muscle relaxation depth from a first-person perspective without leaving the surgical field of vision. This significantly improves the continuity of intraoperative monitoring and the efficiency of collaborative decision-making. Therefore, it solves the problem of high perception delay and low collaborative efficiency caused by the separation of muscle relaxation status monitoring information from the surgical field of vision, achieving the effects of reducing perception delay, improving coordination efficiency, and enhancing surgical safety. Attached Figure Description

[0019] Figure 1 This is a structural block diagram of an AR-based visual guidance system for muscle relaxation states according to an embodiment of the present invention; Figure 2 This is a flowchart of an AR-based visual guidance method for muscle relaxation state according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the adaptive brightness adjustment curve according to Embodiment 3 of the present invention; Figure 4 This is a simulation diagram of virtual layer space stability verification according to Embodiment 6 of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0021] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0022] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0023] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0024] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0025] like Figure 1 As shown, this embodiment provides an augmented reality-based visual guidance system for muscle relaxation, including a muscle relaxation monitoring module, an image perception module, an edge processing module, and an optical fiber augmented reality terminal. This system establishes a real-time, spatially perceptive feedback loop between the physical surgical space and the patient's electrophysiological data stream, enabling the precise anchoring of the patient's muscle relaxation depth index within the surgical field of view in the form of a virtual geometric layer. Furthermore, the system's operating logic does not rely on the user's qualitative judgment but is based on a deterministic coordinate transformation matrix and a nonlinear mapping algorithm. This solves the perception delay problem caused by the separation of monitoring data and the surgical operation space in existing technologies, achieving the technical effect of improving the accuracy of surgical team collaboration and the real-time monitoring of physiological states. It can also transform complex physiological values ​​into intuitive visual information bound to the patient's body space, allowing medical staff to monitor the patient's muscle relaxation state in real time without interrupting the current surgical procedure, either through peripheral vision or by focusing on virtual information, thereby improving the safety and smoothness of the surgical process.

[0026] The muscle relaxation monitoring module is a dedicated medical electronic device that includes the following main components: Electrical stimulation generator: Employs a constant current source circuit controlled by a high-precision digital-to-analog converter (DAC). This constant current source can generate biphasic or monophasic square wave current stimulation within a pulse width of 10ms ± 1ms with a step accuracy of 1mA. The stimulation frequency is programmable, typically set to 2Hz (i.e., one pulse every 50ms) in TOF mode.

[0027] Electromyography (EMG) sensor: Employing a pair of Ag / AgCl surface electrodes, such as those attached to the motor points and tendons of the adductor muscle of the thumb, to acquire microvolt-level electrophysiological signals generated by muscle contraction. The sensor incorporates a front-end analog circuitry, including a low-noise amplifier (LNA) and a bandpass filter (e.g., 10Hz to 500Hz) to remove power line interference and baseline drift. The signal sampling rate is set to 1000Hz, with a resolution of 24 bits.

[0028] Microcontroller (MCU): Integrates a 32-bit ARM Cortex-M4 architecture processor with a working frequency of 200MHz. The MCU is responsible for controlling the timing and current output of the stimulation generator, processing the analog-to-digital converter (ADC) of EMG signals, executing the peak detection algorithm for EMG signals, calculating the TOF ratio, and managing the wireless communication protocol stack. The MCU has 512KB of built-in flash memory and 128KB of RAM.

[0029] Wireless communication module: Employs a Bluetooth Low Energy (BLE 5.0) module, supporting a data throughput of 1Mbps to ensure that muscle relaxation data is transmitted to the image processing module with a latency of less than 5ms. This module also supports GATT (Generic Attribute Profile) service, providing a standardized data interface.

[0030] Data acquisition and processing flow: 1. After receiving the start command from the augmented reality system, the MRMU begins executing the stimulus sequence. It sends the first square wave stimulus pulse, lasting 10ms, with a current of 50mA.

[0031] 2. The EMG sensor acquires compound action potential (CMAP) signals within a window of 20ms to 100ms after stimulation. The MCU executes a real-time peak detection algorithm to identify and record the peak amplitude of the CMAP signal. (unit: The peak detection is achieved by finding local maxima in the signal and combining them with a threshold determination mechanism.

[0032] 3. After a 500ms interval, the MRMU automatically repeats steps S710-S720 three times to acquire data sequentially. .

[0033] 4. The MCU follows the formula Calculate the ratios of the four cascaded stimuli; to ensure the validity of the data, if Too small (e.g.) This indicates that the stimulation is ineffective or the electrode contact is poor. Too large (e.g.) If the signal quality is poor (indicating that the muscle is in a completely unblocked state or that artifacts are present), the data for that round is discarded and stimulation is automatically repeated.

[0034] 5. Calculate the The numerical value (e.g., 0450 ± 0.005) is encapsulated into a 16-byte BLE data packet and broadcast at 100ms intervals. The data packet contains... Value, timestamp, serial number, and CRC checksum.

[0035] The image perception module is integrated into the front end of the augmented reality glasses. Its main task is to provide accurate visual input and motion tracking data to construct real-time six-degrees-of-freedom (6DoF) pose estimation, specifically including: The binocular camera module employs two global shutter color image sensors with a resolution of 1280×720 pixels and a frame rate of 90Hz. The camera module spacing is 65mm (simulating the human eye baseline), and the field of view is 80°×60°. The global shutter avoids the rolling shutter effect that occurs during fast movement, ensuring image sharpness in dynamic scenes. Each camera is equipped with automatic exposure and automatic white balance functions to adapt to the complex lighting environment of the operating room.

[0036] Inertial Measurement Unit (IMU): Integrates a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, with a sampling rate of 1000Hz. The IMU provides low-latency attitude (pitch, yaw, roll) and position information through extended Kalman filtering (EKF) or complementary filtering algorithms. The gyroscope accuracy is ±0.1° / s, and the accelerometer accuracy is ±0.1m / s².

[0037] Depth sensor (optional): To enhance spatial awareness and depth estimation in textureless regions, the system may be equipped with a time-of-flight (ToF) or structured light depth sensor. It provides a depth map of the scene at a frequency of 30 Hz, with a resolution of 320 × 240 pixels and an effective range of 0.5 m to 3 m. This depth information can be used to assist in scale estimation and scene understanding in SLAM algorithms.

[0038] Ambient light sensor: The sensor uses a high dynamic range (HDR) photodiode with a measurement range of 1 lux to 100,000 lux to sense ambient illuminance in real time to support the display's adaptive brightness adjustment function.

[0039] The image processing module is responsible for all complex calculations, data fusion, and rendering instruction generation. It can be a standalone edge computing box, attached to a doctor's waist or neck, or it can be highly integrated into the temples of augmented reality glasses to achieve a lightweight design. Specifically, it includes: Hardware platform: It adopts a low-power, high-performance system-on-a-chip (SoC), such as a multi-core processor based on the ARM architecture (e.g., a quad-core A78 processor), integrating a graphics processing unit (GPU) and a neural network processing unit (NPU); the processor has a main frequency of 2.5GHz and 8GB of LPDDR5 memory. The SoC has hardware-accelerated video decoding and encoding capabilities.

[0040] Its software framework is as follows: Operating System: Based on a real-time Linux kernel, task scheduling and interrupt handling are optimized to ensure low-latency execution of critical tasks such as pose tracking and rendering.

[0041] The SLAM algorithm library implements a visual inertial odometry (VIO) algorithm, fusing data from binocular cameras and IMUs to output high-precision 6DoF pose at a frequency of 90Hz. Simultaneously, it maintains a sparse feature point map for loop closure detection and global pose optimization, ensuring robustness over long periods of operation.

[0042] Rendering Engine: Based on the Vulkan or OpenGL ES graphics API, it achieves efficient virtual object rendering. It supports multi-layer rendering, depth testing, lighting models, and real-time post-processing effects (such as anti-aliasing and tone mapping) to generate high-quality virtual information layers.

[0043] Data fusion and analysis module: responsible for receiving muscle relaxation monitoring data (from the BLE module), performing filtering, outlier removal, trend prediction, etc.; in addition, this module can also integrate other physiological data (such as electrocardiogram, pulse oximetry, etc.) for composite risk assessment.

[0044] Collaborative Communication Module: Manages wireless connections (e.g., Wi-Fi 6) with multiple augmented reality glasses and a central monitoring system, enables the synchronous distribution of data and rendering instructions, and handles the encryption and decryption of data packets.

[0045] Visual Data Integrity Monitoring Unit: This unit is used to ensure the reliability of augmented reality display information.

[0046] Augmented reality display terminals represent the ultimate human-computer interaction interface. Their design takes into account the specific needs of the medical environment (such as lightweight design, comfort, ease of disinfection, and high transmittance), specifically including: Microdisplays: Employ two 0.39-inch LCoS (Liquid Crystal on Silicon) or Micro-LED microdisplays with a resolution of 1920×1080 pixels, a brightness of 3000 nits, and a contrast ratio of 100000:1. Supports full-color display (16.7 million colors). The microdisplays feature a fast response time (less than 5ms) to reduce motion blur.

[0047] Optical waveguide engine: Input coupler: The image beam from the microdisplay is guided into the waveguide sheet via a microprism array or holographic optical element in a total internal reflection manner.

[0048] Waveguide sheet: Made of high-refractive-index glass or polymer material (e.g., lithium niobate crystal with a refractive index of 1.9), its surface is etched with a micro / nano structured grating. The grating is precisely designed to reflect image light multiple times internally and propagate along the waveguide sheet, achieving uniform light transmission. The waveguide sheet is 1.5 mm thick.

[0049] Exit pupil expander: A special grating structure at the end of the waveguide plate diffracts the light propagating inside evenly out of the waveguide plate, forming a wide exit pupil (e.g., 10mm×10mm), ensuring that doctors can see a complete virtual image even without precisely aligning the pupil.

[0050] Output coupler: Directs the output of image light to the human eye while maintaining 85% transmittance to ambient light, ensuring that doctors can clearly observe the actual surgical area.

[0051] Wearing and Comfort: The total weight of the glasses is less than 100g (excluding the calculation unit). The frame features an adjustable design to ensure a comfortable fit for doctors with different head shapes. The contact surfaces are made of medical-grade silicone or biocompatible materials, and support high-temperature and high-pressure sterilization or ultraviolet sterilization, meeting medical and health standards.

[0052] Field of view (FOV): 45° horizontal field of view, 30° vertical field of view. The virtual information is displayed without interruption in the surgical area where the surgeon is primarily focused, ensuring the integrity of the information.

[0053] The system implements an AR-based visualization-guided method for muscle relaxation, such as... Figure 2 As shown, the method includes the following steps: Step S100: Obtain real-time muscle relaxation monitoring data of the patient collected by the muscle relaxation monitor, wherein the real-time muscle relaxation monitoring data includes four series of stimulation ratios.

[0054] In this embodiment, by employing an asynchronous serial communication bus between the muscle relaxation monitoring module and the image processing module, it is possible to continuously acquire the muscle contraction response vector of the patient after stimulation.

[0055] The muscle relaxation monitor, acting as a slave device, periodically sends raw data packets containing electrophysiological indicators to the image processing module, which acts as the master device. The structure of the raw data packet is defined as a fixed-length byte stream, which includes a start frame, an instruction type field, a data payload length, a muscle relaxation monitoring value field, a timestamp field, and a cyclic redundancy check field. The muscle relaxation monitoring value field stores four chained stimulation ratios calculated internally by the muscle relaxation monitor. The four chained stimulation ratios are set as the arithmetic ratio of the fourth contraction peak to the first contraction peak, and the value range is limited to the closed interval [0, 1.0].

[0056] To address potential signal abrupt changes and transient data anomalies caused by the complex electromagnetic environment within the operating room (such as electrosurgical units and high-frequency radio frequency devices), after analyzing four consecutive stimulation ratios, an outlier detection operation based on the three-standard-deviation criterion is enforced. The system maintains a temporary observation queue of length 20. By calculating the mean and variance of historical data in this queue in real time, it determines whether the currently acquired value belongs to abnormally fluctuating electromagnetic noise or sensor artifacts. Specifically, if the deviation of the current value from the queue mean exceeds three standard deviations, the value is marked as an outlier. If an outlier is identified, the system does not directly use the outlier value but instead uses the previous valid sample value for linear compensation or replaces it with the queue median to maintain the continuity and stability of the data stream.

[0057] For example, in a specific clinical monitoring scenario, the muscle relaxation monitor sends four consecutive pulse stimulations at a frequency of 2Hz via ulnar nerve electrodes. The system then acquires a sequence of raw muscle contraction intensity data, with the following values: ,as well as (characterized by electromyographic current amplitude), then the parsing unit in the image processing module first uses the formula Perform the initial operation to obtain The system then stores this value in a preset sliding window buffer, with the window size W set to 10 sampling points. Assume the historical sequence already existing in this buffer is... To achieve a smooth visual representation, the system uses a weighted average algorithm to calculate the final ratio of the four cascaded stimuli. Its weight allocation follows a Gaussian distribution model, assigning higher weights to values ​​most recent in time to reflect their higher real-time priority. The Gaussian weight function... In the formula, i is the index within the window, and Q is the window size. It is the standard deviation, which is set to W / 4 for example.

[0058] Assuming the weights within the window are normalized, the current value of 0.300 is assigned a weight of 0.4, and the total weight of the remaining 9 historical averages is 0.6. Thus, the smoothed... The value of 0.2994 was immediately timestamped in microseconds and passed as input to the subsequent rendering engine, thus ensuring that the system could obtain a stable muscle relaxation depth index within a 100ms sampling period, providing a high-confidence data source for subsequent virtual layer rendering. Moreover, through this dual preprocessing (outlier removal and weighted moving average filtering), false positive or false negative alarms caused by measurement noise or random interference were significantly reduced, thereby improving the clinical usability and safety of the system.

[0059] In step S200, the image processing module is used to capture image information of the surgical field of view, and a real-time surgical space coordinate system containing the physically anchored target is constructed based on the image information.

[0060] In this embodiment, by using the binocular vision sensor built into the augmented reality glasses, the three-dimensional geometric features of the surgical site can be extracted in real time, and a real-time surgical space coordinate system can be constructed.

[0061] Specifically, this includes a nested coordinate transformation system: the camera coordinate system of augmented reality glasses. World Reference Coordinate System and physical anchoring coordinate systems specific to patient body parts. The system maintains the relative pose matrix between these coordinate systems in real time by executing the Simultaneous Localization and Mapping (SLAM) algorithm. The SLAM algorithm adopts the Visual Inertial Odometry (VIO) framework, which integrates high-frequency image feature point data from binocular cameras and gyroscope / accelerometer data from the inertial measurement unit (IMU).

[0062] In the process of constructing the coordinate system, the image processing module first performs multi-scale feature point detection and descriptor extraction on each captured frame of image. The feature points are selected from objects in the surgical field of view that have high contrast, high texture and spatial stability, such as the edge seams of the surgical drape, the metal corners of the fixation instruments, or markers with specific geometric patterns that are pre-attached to the patient's body surface. Here, the system uses ORB (Oriented Fast and Rotated BRIEF) or SIFT (Scale-Invariant Feature Transform) to encode these feature points, and establishes a correspondence between consecutive frames through a feature matching algorithm accelerated by kd-tree.

[0063] To achieve anchoring accuracy down to the millimeter level, when the system identifies a pre-defined physical anchoring target (such as an AR marker with known geometric dimensions) in an image, it utilizes its known geometric dimensions and pixel position in the binocular image to execute a monocular or binocular Perspective-n-Point (PnP) pose estimation algorithm to calculate the transformation matrix from the camera center to the anchoring target center. The matrix contains a 3×3 rotation matrix R and a 3×1 translation vector t, and anchors the physical coordinate system. Transformation of points to camera coordinate system .

[0064] For example, during a laparoscopic surgery under general anesthesia, the anesthesiologist wears augmented reality glasses and faces the patient. The system uses the left-side camera to capture a pre-placed square physical marker located to the side of the patient's shoulder. The physical side length L of this marker is 50.0 mm. At this time, the image processing module detects the pixel coordinates of the four corner points of this marker in the left-eye image coordinate system. The system then loads the intrinsic parameter matrix from the left eye camera. (Unit: pixels, where 1080 is the focal length, and (960, 540) is the principal point); At this point, the translation vector t and rotation matrix R of the camera relative to the marker are obtained; assuming the calculation result is... This means that the doctor's current observation point (camera optical center) is located 120.5 mm to the right, 45.2 mm above, and 650.0 mm away from the center of the marker in the world coordinate system. At the same time, the system calculates the rotation matrix R, which determines the pitch, yaw, and roll angles of the line of sight. Here, it is assumed that the Euler angles of R are [15°, -5°, 0°].

[0065] This allows us to obtain the real-time surgical space coordinate system. Subsequently, the system defines the center of the physical landmark as the world coordinate system. The origin is (0,0,0), and so on; all subsequent virtual information layers, such as the muscle relaxation depth dashboard, will be preset at offset coordinates relative to this origin. The coordinate system is constructed based on real-time image analysis, combined with high-frequency IMU data for motion prediction and compensation. This ensures that even if the anesthesiologist moves around in the operating room or turns his head significantly, the virtual dashboard can still be visually locked in a fixed position near the patient's body. Its spatial drift error has been verified in the laboratory to be controllable within 3mm.

[0066] Step S300: Input the four series of stimulus ratios into a preset visual mapping model to generate a corresponding virtual information layer, wherein the virtual information layer includes color features and graphic geometric features associated with the values ​​of the four series of stimulus ratios.

[0067] In this embodiment, the visual mapping model is configured to have multi-dimensional output capabilities, including color gradation, dynamic geometric transformation, and text numerical display, to adapt to different doctors' visual preferences and information interpretation habits. Specifically, it includes a non-linear color space conversion function, which is constructed based on preset physiological thresholds and safety ranges. The system first obtains the smoothed... The numerical value is input into a double-threshold piecewise function, which will... The interval is divided into three main areas: the safe zone (e.g., the safe zone) ), transition zone (e.g. ) and warning areas (e.g. Each region corresponds to a different color gradient strategy.

[0068] The conversion to visual output is achieved through a non-linear weighted formula used to calculate the rendered color vector, which defines the target rendered color. The vector is a three-dimensional vector (R, G, B). To achieve a smooth and physiologically meaningful transition from dark green to bright red, a hybrid interpolation strategy based on the Sigmoid function can be used. The Sigmoid activation function... Used to map an input value to the interval (0,1):

[0069] in, It is a dimensionless muscle relaxation ratio value acquired in real time. The threshold value for the midpoint of the color transition can be set to 0.45, representing the physiological critical point for muscle relaxation from deep to light. k is the sensitivity slope parameter, which can be set to 25. This parameter controls the steepness of the color transition curve, ensuring rapid and significant color change near the critical threshold to improve the immediacy of visual alertness. A larger k value indicates that the color change is more pronounced at the critical threshold. The changes are steeper in the vicinity, while smaller k values ​​make the transitions smoother, and so on; The value is a weighting factor between 0 and 1, when far below hour, A value close to 0 indicates a state of deep muscle relaxation; when... Much higher hour, A value approaching 1 indicates a warning state of insufficient muscle relaxation; the system uses this as a basis for its actions. Value in alarm color vector (Pure red represents high risk) and safety color vector Linear interpolation is performed between (pure green, representing safe depth of muscle relaxation) to calculate the final rendered color:

[0070] This ensures that the color is greenish when the muscles are sufficiently relaxed and reddish when they are insufficiently relaxed. This continuous gradient from green to yellowish-green, amber, orange and then to red provides doctors with intuitive and physiologically relevant visual feedback, avoiding misinterpretations that may be caused by discrete colors.

[0071] For example, suppose the current time has been smoothed. The value is 0.65, and the system substitutes the parameter into... function: (here) The closer the value is to 1, the closer the state is to... (i.e., red), the system then calculates the target rendering color. (After rounding) In the augmented reality display terminal, the virtual information layer will be displayed as a striking, highly saturated orange-red block with an RGB value of (253,2,0), which intuitively indicates to the anesthesiologist that the current muscle relaxation state is in a significantly insufficient warning range.

[0072] In addition to color features, the virtual information layer also includes dynamically changing graphic geometric features. In this embodiment, a semi-circular virtual dashboard is constructed, whose appearance design simulates the visual habits of traditional pointer instruments in order to utilize the existing cognitive models of medical staff. The dashboard is rendered in a three-dimensional virtual space using computer graphics technology and is designed to be semi-transparent to avoid obstructing the real field of vision.

[0073] Dashboard Background and Gradient Generation: The dashboard background is a semi-circular plane with a diameter D of 100mm, and its material is set to a semi-transparent frosted glass effect (e.g., 50% transparency) to avoid obstructing the doctor's observation of the actual surgical environment; the scale lines and numerical labels (e.g., from 0.0 to 1.0) are evenly distributed along the semicircle, with a scale interval of 0.1; the scale lines are generated using a procedural method, i.e., by calculating the coordinates of discrete points on the semicircle and drawing white line segments with a length of 5mm and a width of 0.5mm; and to further enhance visual recognition, in A thick red warning line, 1.5mm wide, is placed at the location, and... A thick green safety line, 1.5mm wide, is set at the location.

[0074] The core geometric feature of the instrument panel is its pointer, which is a rectangular geometric shape 45mm long and 2mm wide, with its center of rotation located at the center of the semicircle of the instrument panel; the pointer's rotation angle θ (unit: degrees) is... A linear mapping of values; to ensure When the value is 0.0, the pointer points to the left endpoint of the semicircle (i.e., 0°), and When the value is 1.0, it points to the right endpoint (i.e., 180°), and the mapping function is:

[0075] in, For initial angle compensation, it is exemplarily set to 0°, such that Corresponding to 0°, Corresponding to 180°.

[0076] To make the pointer movement smoother and more natural, the rendering engine uses a low-pass filter or damping algorithm for pointer rotation. This algorithm tracks the difference between the pointer's instantaneous angular velocity and the target angular velocity, and applies an exponentially decaying function to adjust its angular acceleration, avoiding abrupt jumps and thus reducing visual fatigue and interference. For example, if the target angle is... The current angle is Then update the angle. ,in It can be set to 0.1.

[0077] Numerical text display: Precise numerical values ​​are displayed in real-time in a specific font (e.g., Sans-serif font, 18pt size) in the center or below the dashboard. Numerical values ​​should be rounded to two decimal places; the text color should match the color of the dashboard pointers to achieve visual consistency, as shown above. When the text is displayed in orange-red, the text will be displayed in red, and so on.

[0078] Warning icons and animations: When When the warning threshold of 0.70 is exceeded, in addition to changes in color and pointer, a dynamically flashing red triangle warning icon will pop up next to the dashboard, accompanied by a soft visual pulse animation to further attract the doctor's attention; the flashing frequency of the icon is exemplarily set to 2Hz for 5 seconds, and its transparency changes periodically between 50% and 100%.

[0079] For example, following the aforementioned The value.

[0080] Color determined: Calculated as (253,2,0), i.e., orange-red. This color will be applied to the dashboard pointers, numerical text, and alert icons (if triggered).

[0081] Pointer angle calculation: θ = 0.65 × 180° = 117°. The rendering engine will draw an orange-red pointer extending from the center of the circle, pointing in the 117° direction. If the pointer angle in the previous frame was 110°, the damping algorithm will slowly and smoothly move it towards 117°.

[0082] Text display: Below the center of the dashboard, the rendering engine will display "0.65" in orange-red font.

[0083] Warning icon triggering: Since 0.65 < 0.70, the flashing warning icon will not be triggered at present. Through this multimodal visual feedback mechanism, doctors can not only intuitively judge the muscle relaxation state from the color, but also obtain precise quantitative information from the pointer angle and specific values. At the same time, the dynamic animation enhances the sense of hierarchy of the warning, ensuring that key physiological information can be quickly and accurately identified and understood in a highly stressful surgical environment.

[0084] Step S400: Calculate the projection parameters of the virtual information layer in the augmented reality glasses display unit based on the real-time surgical space coordinate system; In this embodiment, a multi-level projection transformation pipeline based on computer graphics is used to accurately map the three-dimensional virtual information layer onto the two-dimensional display plane, ensuring spatiotemporal consistency with the physical world.

[0085] The calculation of projection parameters involves converting the world coordinate system vertices of the virtual information layer into screen pixel coordinates for the augmented reality glasses through a series of matrix operations. This process strictly follows the standard workflow of the computer graphics rendering pipeline, including model transformation, viewpoint transformation, and projection transformation. Model transformation: Transform the vertices in the local coordinate system of the virtual dashboard. Transform to world coordinate system This involves the preset position and orientation of the virtual dashboard relative to the physically anchored target. The dashboard is modeled in the local coordinate system as a semi-circular grid located at the origin (0,0,0); for example, the local center point coordinates of the virtual dashboard are... In the world coordinate system, we expect it to be located 200mm above the physical landmark on the patient's shoulder. For easier observation by the doctor, it is offset by 50mm towards the doctor (along the negative Z-axis of the world coordinate system) and rotated 15° around the X-axis (i.e., the horizontal axis) to make it slightly tilted towards the doctor. The model transformation matrix is ​​then... It will consist of a 200mm translation along the Y-axis, a -50mm translation along the Z-axis, and a -15° rotation around the X-axis: .

[0086] Through this transformation, the virtual dashboard acquires a fixed spatial position and orientation in the world coordinate system; for example, its center point becomes... This position is constant relative to the patient's body; it will remain in that relative position no matter how the doctor moves it, and so on.

[0087] Viewpoint transformation: Transforms virtual objects in the world coordinate system. Transform to camera coordinate system This is determined by the real-time pose of the augmented reality glasses, which is derived from the rotation and translation transformation matrix of the camera relative to the world coordinate system. The inverse matrix, i.e. For example, if the aforementioned transformation matrix of the camera relative to the world is Then the matrix that transforms world coordinates to camera coordinates is: This ensures that no matter how the doctor moves their head, the virtual object can be correctly positioned relative to the doctor's viewpoint, maintaining its relative position in the physical world.

[0088] Projection transformation: transforming points in the 3D camera coordinate system Mapped to two-dimensional normalized device coordinates (NDC). Then it is converted to screen pixel coordinates, which can be done using the camera's intrinsic parameter matrix K (including focal length). and the main point This is accomplished by simulating the perspective effect observed by the human eye; for example, the camera intrinsic parameter matrix. Assuming that after viewpoint transformation, the coordinates of a vertex of the virtual dashboard in the camera coordinate system are... Then its pixel coordinates (u,v) on the image plane are calculated as follows:

[0089] Where u and v are image coordinates in pixels, representing the projection position of the three-dimensional point on the two-dimensional screen.

[0090] It should be noted that since augmented reality glasses typically employ complex optical systems (such as optical waveguides, freeform prisms, etc.), these optical components introduce various geometric distortions, such as radial distortion (barrel or pincushion) and tangential distortion. In order to ensure that the virtual information layer accurately overlaps with the physical environment, an inverse distortion correction algorithm is forcibly executed after the projection transformation. This algorithm is based on the distortion parameters obtained in advance through camera calibration.

[0091] Specifically, the inverse distortion correction algorithm is based on pre-calibrated distortion parameters. (Radial distortion coefficient) and (Tangential distortion coefficient); for example, assuming the ideal (distortion-free) image point at pixel coordinates (u,v) is The actual observed distortion points are The system calculates the radial distortion factor. Corrected ideal point coordinates:

[0092]

[0093] Through precise inverse distortion correction, the system ensures that the straight edges of the virtual dashboard remain straight in the doctor's field of vision, without barrel or pincushion distortion, thus significantly improving the realism of virtual-real fusion and user experience. Of course, in actual rendering, the mapping from the ideal rendering position to the actual display position is achieved through reverse lookup tables or iterative solutions.

[0094] Furthermore, in order to generate a virtual information layer with a sense of depth in binocular augmented reality glasses, the system calculates a slightly different set of projection parameters for each eye.

[0095] For example, for the left eye, the screen coordinates are calculated following the steps described above. For the right eye, since the camera position has a horizontal offset B (baseline distance, exemplarily 65mm) relative to the left eye, the system will take into account the relative position of the right eye camera during the viewpoint transformation stage, and calculate another transformation matrix to obtain... These minute pixel differences (parallax), after being superimposed on the retinas of the left and right eyes, are fused by the brain to form a stereoscopic perception of the virtual information layer. For example, when a virtual dashboard is anchored 500mm from a doctor's eyes, the horizontal parallax of its left and right eye images will precisely correspond to the stereoscopic effect that should exist at that physical distance. The parallax calculation formula is... Where f is the focal length and B is the baseline. By representing the depth of objects, this processing not only enhances the sense of immersion but also allows doctors to more accurately determine the depth position of virtual layers in the surgical space, thereby achieving more natural interaction and perception.

[0096] In step S500, augmented reality glasses are used to overlay and project virtual information layers onto a preset position in the surgical field of view to achieve immersive monitoring of the patient's muscle relaxation state.

[0097] In this embodiment, by employing high-resolution augmented reality glasses that integrate waveguide display technology, the virtual information layer generated by the image processing module is seamlessly integrated into the doctor's observation of the real surgical area. The augmented reality glasses serve as the display terminal, and their display module is equipped with a microdisplay (e.g., a liquid crystal on silicon (LCoS) microdisplay or a Micro-LED array) and an optical waveguide engine. The microdisplay is responsible for converting the digital image signal from the image processing module into a high-brightness, high-contrast grating image, while the optical waveguide engine projects these grating images directly onto the doctor's retina in the form of a parallel beam through internal total internal reflection and diffraction mechanisms, while maintaining extremely high transmittance to external ambient light.

[0098] To ensure accurate and stable visual overlay between the virtual information layer and the physical environment, the following mechanisms are implemented: 1. The display refresh rate of the augmented reality glasses is set to 90Hz or higher (e.g., 120Hz), and the end-to-end latency of the rendering pipeline (from sensor data acquisition to final display) is strictly controlled within 20ms to reduce motion blur or ghosting of virtual objects during rapid head movements by the doctor, ensuring high synchronization between virtual information and the real world. For example, if the rendering latency is 15ms, and the doctor's head rotates at an angular velocity of 100° / s, then within those 15ms, the head actually rotates 1.5°. Without compensation, the virtual object will visually lag behind by 1.5°, creating a "drifting" effect. Asynchronous time warping... The system uses the Timewarp (ATW) or SpaceWarp algorithm to predict the head's pose changes within a 15ms rendering cycle using the aforementioned IMU data. After the image rendering is completed and before the display scans and outputs the image, a 1.5° inverse geometric transformation is performed on the final image. This inverse transformation is executed in a GPU hardware-accelerated manner, taking less than 1ms. This reduces the visual lag of the virtual object to below the human perception threshold (<5ms), making the virtual layer appear to always be "attached" to the physical world.

[0099] 2. The lighting environment in the operating room is complex and dynamically changing (e.g., the on / off state or brightness adjustment of the shadowless lamps). To ensure that virtual information is clearly visible under any lighting conditions without causing visual interference, augmented reality glasses integrate an ambient light sensor. The ambient light sensor (e.g., a photodiode array) samples the ambient brightness at a frequency of 50Hz. Subsequently, the adaptive adjustment module in the image processing module dynamically adjusts the output brightness of the microdisplay and the contrast and saturation of the rendered image based on the real-time brightness value. When the ambient brightness is detected to increase sharply from 800 lux (general indoor lighting) to 50,000 lux (shadowless lamps at full power), the system can detect the change in ambient brightness. When the system is activated, it will linearly increase the brightness of the microdisplay from 500 nits to 1800 nits within 200ms. At the same time, the contrast of the virtual information layer will be adjusted according to a preset gamma curve to avoid "washout" in high-brightness backgrounds. For example, when the ambient brightness exceeds 40,000 lux, in order to enhance the visual impact of warning colors (such as red), the color saturation will be slightly increased by 10%, and the relative ratio of the brightness of the text color to the background will be adjusted. This ensures that the virtual layer remains visible in scenes with a wide dynamic range of brightness and minimizes interference with the doctor's natural vision.

[0100] 3. Due to differences in interpupillary distance (IPD) and pupil position among different doctors, the optical system design of augmented reality glasses incorporates an exit pupil expansion function to ensure that users can see the complete virtual image without precisely aligning their pupils. Furthermore, the system monitors the user's pupil position in real time using eye-tracking sensors, fine-tuning the center of the displayed image to optimize edge sharpness. For example, eye-tracking sensors (including an infrared-based camera) capture the doctor's gaze points at a frequency of 120Hz. If the system detects that the doctor's pupil deviates 2mm from the optical exit pupil center, it uses digital beam shifting technology to perform a small displacement compensation on the microdisplay, ensuring that the center of the virtual image is always aligned with the center of the doctor's pupil. This maximizes image sharpness and field of view utilization. This pupil centering calibration can control the visual blur area within a 0.5° field of view angle, ensuring the clarity of key information. Simultaneously, the large exit pupil design (e.g., 10mm × 10mm) provides wearing flexibility, reducing the need for repeated adjustments.

[0101] 4. Real-time muscle relaxation monitoring data and projection parameters are synchronized to multiple augmented reality glasses via wireless communication protocols, enabling anesthesiologists and surgeons to share the same visual information on muscle relaxation status; for example, the image processing module acts as a central synchronization server, broadcasting muscle relaxation data updates to all registered augmented reality glasses in the network via encrypted Wi-Fi 6 protocol (e.g., The system incorporates color and geometric layer parameters, along with corresponding spatial anchor points. Each receiving glasses independently calculates the projection of the virtual information layer onto its own field of view based on its real-time pose and calibration parameters. This ensures that anesthesiologists and surgeons (if the surgeon also wears AR glasses) can see a completely consistent muscle relaxation dashboard that is spatially anchored in the same physical location from their respective perspectives. For example, when a surgeon is concentrating on laparoscopic surgery, the system can immediately detect the muscle relaxation dashboard located 20cm above the patient's abdomen, displayed in orange-red with the pointer pointing to 0.65. This allows for instantaneous synchronization of muscle relaxation status without any verbal communication, enabling seamless collaborative decision-making and significantly improving the overall efficiency and response speed of the surgical team.

[0102] Example 2 The difference from Example 1 is that this example no longer relies solely on the first-order derivative to determine instantaneous changes, but instead employs a muscle relaxation trend prediction model based on a multi-order Kalman filter (EKF), by fusing historical observation data ( The system uses a sequence of data and a dynamic model (a pharmacokinetic model of muscle relaxants) to optimally estimate the future state of the muscle relaxant ratio. This involves treating the change in the muscle relaxant ratio as a noisy nonlinear dynamic system and continuously refining the estimates of the system state (such as the current value, first-order rate of change, and second-order acceleration). At this point, the system state vector... ,in The muscle relaxation ratio at time k. Its rate of change, Given its acceleration, the system dynamic model is defined as follows: Where f is a nonlinear state transition function (e.g., considering the drug dosage). ), For process noise; Measurement model: Where h is a nonlinear measurement function, To measure noise, EKF recursively estimates the state by linearizing the nonlinear function.

[0103] For example, suppose the system uses a fixed time step of 10 seconds. To predict the muscle relaxation ratio, at a certain moment, the EKF estimates the current system state as follows: At this point, the system needs to predict the future. The muscle relaxation ratio will be determined by iteratively applying the linearized state transition matrix. about Completed in one step; predicting the state muscle relaxation ratio component It can be approximated as:

[0104] Substitution ; That is, after 60 seconds, the muscle relaxation ratio may rise to 0.730, which exceeds the warning threshold of 0.70.

[0105] In addition, the augmented reality glasses display a solid green pointer indicating the current depth of muscle relaxation on the muscle relaxation depth dashboard. They also generate a semi-transparent, slowly pulsating orange-red prediction pointer pointing to the 0.730 position. The pulsation frequency of the prediction pointer can dynamically change with the prediction uncertainty (determined by the diagonal elements of the covariance matrix of the Kalman filter). The higher the uncertainty, the faster the pulsation (for example, a 10% increase in prediction error variance results in a 1Hz increase in pulsation frequency). This prediction information non-invasively alerts the anesthesiologist that if the current trend continues and no additional muscle relaxants are administered, the patient will be at serious risk of insufficient muscle relaxation depth after 1 minute, and so on.

[0106] Example 3 To ensure that the virtual information layer maintains optimal visibility and contrast under the extremely wide dynamic range of lighting conditions in the operating room, and to avoid information becoming difficult to identify due to drastic changes in ambient brightness, this embodiment utilizes an ambient light sensor on the augmented reality glasses to perceive the illuminance of the surrounding environment in real time.

[0107] Specifically, such as Figure 3 As shown, based on the ambient illuminance value, the system dynamically adjusts the brightness, contrast, and local tone mapping parameters of the virtual information layer. This is specifically achieved through a brightness-contrast mapping curve, which maps the ambient illuminance value to the display's output brightness range using a non-linear function (such as a logarithmic or exponential function). The brightness adjustment function is as follows:

[0108] Ensure that the brightness increases linearly with the logarithm of the ambient illuminance.

[0109] The contrast adjustment function is:

[0110] The squared term enhances contrast under high illumination. Minimum / maximum brightness of the display (unit: nits); The current ambient illuminance (unit: lux) The maximum ambient illuminance supported by the system (unit: lux); Reference contrast; This is the contrast enhancement factor, which is dimensionless.

[0111] For example, the minimum brightness of the microdisplay in augmented reality glasses. Maximum brightness The system supports a maximum ambient illuminance. Reference contrast Contrast enhancement factor .

[0112] Scene 1: Localized lighting, illuminance .

[0113]

[0114] At this point, the virtual layer has moderate brightness and slightly higher contrast than the baseline, avoiding excessive brightness in local dark areas while ensuring clarity.

[0115] Scene 2: All shadowless lights on, illuminance .

[0116]

[0117] At this point, the virtual layer is displayed at near maximum brightness, significantly improving contrast and ensuring that muscle relaxation information (especially warning colors) remains clear and conspicuous even in extremely bright surgical fields, avoiding the risk of information being overwhelmed. And so on.

[0118] Example 4

[0119] The metabolism of muscle relaxants, individual physiological differences in patients, and surgical stimulation all affect muscle relaxation. Therefore, this embodiment, in addition to... In addition, heart rate variability (HRV) was introduced as an auxiliary assessment indicator to reflect the activity of the autonomic nervous system (ANS): when patients have insufficient muscle relaxation and experience painful stimuli, the sympathetic nervous system is activated, and HRV changes significantly (e.g., a decrease in the high-frequency component and an increase in the low-frequency / high-frequency ratio). This was achieved by establishing... The system identified the correlation model between HRV and HRV. The model identifies the risk of latent muscle relaxation deficiency, where the patient may actually be experiencing stimulation (such as visceral traction) but is at a borderline level. Additionally, the model employs a Bayesian network or logistic regression model to... Using HRV (Heart Risk Value) as input, the model outputs a comprehensive risk score. It is trained offline and built using a large amount of clinical data. The nonlinear relationship between HRV and actual pain / movement risk.

[0120] For example, the system simultaneously obtains The patient's electrocardiogram (ECG) signal (sampling rate 500Hz) is processed using R-peak detection (e.g., Pan-Tompkins algorithm) to calculate continuous RR intervals, thereby obtaining HRV indices, such as the low-frequency / high-frequency power ratio (LF / HFRatio), which is obtained by performing a Fourier transform on the RR interval sequence and calculating the power in the 0.04-0.15Hz (LF) and 0.15-0.4Hz (HF) frequency bands; assuming the input of this model is... .

[0121] Risk Score Set to:

[0122] in, , which are weight parameters obtained through training with clinical data.

[0123] Scene 1: The LF / HF ratio is 1.8 (normal range). At this point, the muscle relaxation ratio is close to the warning threshold, but the autonomic nervous system is relatively calm.

[0124] Risk Score The system displays orange-red. The dashboard has no additional autonomic warnings.

[0125] Scene 2: LF / HF Ratio = 4.5 (sympathetic activation).

[0126] although The same applies, but a high LF / HF ratio suggests that the patient may be experiencing sympathetic excitation due to stimulation or insufficient muscle relaxation. Therefore:

[0127] Risk Score At this moment, the augmented reality glasses are displaying orange-red. In addition to the dashboard, a flashing red text warning, "Autonomic Nervous System Activation: HRV Abnormality," is overlaid in the upper right corner of the field of vision, with a flashing frequency similar to... The correlation is direct, suggesting that in addition to focusing on the muscle relaxation ratio, anesthesiologists should also consider pain stimuli or deeper levels of muscle relaxation insufficiency.

[0128] By integrating multiple physiological parameters, it provides anesthesiologists with a more comprehensive and context-aware assessment of muscle relaxation, effectively avoiding potential misjudgments or delays that may occur with a single indicator, and significantly improving the accuracy and timeliness of clinical decision-making.

[0129] Example 5

[0130] To ensure the reliability of the AR information seen by doctors, the relevant data needs to be verified. This can be achieved by integrating multiple independent physiological monitoring data sources to cross-validate the physiological rationality of the display of muscle relaxation status in augmented reality glasses. This addresses the risk of sensor malfunctions, artifacts, or internal system mapping errors that may result from a single muscle relaxation monitoring data point, leading to incorrect AR display information.

[0131] Specifically, a multimodal physiological data fusion model can be established to continuously receive data from various sources, including... In addition to other key physiological parameters (e.g., heart rate, blood oxygen saturation, end-tidal carbon dioxide partial pressure), the model assesses physiological consistency by analyzing the expected correlation between these physiological parameters and the currently displayed state of muscle relaxation; when the muscle relaxation state displayed by AR deviates significantly from the expected physiological response, the system determines it to be an abnormality of physiological consistency; for example, if AR shows the patient is in deep muscle relaxation ( (Extremely low), but the patient's heart rate and end-tidal carbon dioxide partial pressure show spontaneous breathing activity (rather than complete paralysis), which constitutes an anomaly in consistency, and so on; the model can use a rule-based inference system, or more complex Bayesian networks, anomaly detection neural networks, etc., which are not limited here.

[0132] For example, the system continuously receives the following data: 1. AR displays the current state of muscle relaxation: (Green, deep muscle relaxation).

[0133] 2. Patient's real-time heart rate (from ECG monitor): .

[0134] 3. Patient's blood oxygen saturation (from pulse oximeter): .

[0135] 4. End-tidal carbon dioxide partial pressure of the patient (from the anesthesia machine): .

[0136] Physiological consistency risk scoring model Set as a weighted score based on cumulative bias:

[0137] in, For weights (e.g.) `dev()` is the standardized deviation function. Under normal circumstances, during deep muscle relaxation ( (Very low), the patient should have no spontaneous breathing. It should be very low or zero (or controlled by the ventilator if mechanically ventilated).

[0138] Scenario 1: AR Display (Deep muscle relaxation), but at this time Suddenly rises to 45 mmHg; based on empirical rules, when At that time, the indication of physiological consistency was high; the calculation bias was dev(0.05,45mmHg) which was much higher than the normal threshold.

[0139] The model outputs a physiological consistency risk score. (The threshold is set to 0.75 for example).

[0140] Visual feedback format: due to If the threshold is exceeded, the augmented reality glasses will immediately overlay a bright red rectangle on the right side of the main field of vision, displaying the warning text "AR muscle relaxation information abnormal: inconsistent with EtCO2," accompanied by a 1Hz voice prompt: "Caution, muscle relaxation monitoring data may contain physiological abnormalities." Simultaneously, the AR-displayed muscle relaxation dashboard itself will begin flashing at a frequency of 5Hz, with a red halo around the edges, indicating to the doctor that the currently viewed muscle relaxation data may be unreliable and requires manual verification.

[0141] Example 6 To address the issues of virtual object drift, jitter, or misalignment caused by SLAM system drift, external interference (such as strong reflections or rapid changes in lighting), or physical displacement of AR glasses, and to ensure the physical accuracy of AR displayed content, such as... Figure 4 As shown, the binocular camera of the image perception module can be used to continuously capture the surgical field of view. In each frame of the image, the system not only tracks the physical anchor target (such as the physical markers mentioned above), but also tracks the visual features of the screen area that the virtual information layer should occupy (e.g., the corners or edges of the virtual dashboard). Spatial stability is evaluated by calculating the pixel deviation between the actual rendering position of the virtual information layer on the screen and the expected rendering position inferred based on the physical anchor target. If the deviation continues to exceed a preset threshold for a period of time, it is determined to be an abnormal spatial stability.

[0142] For example, the system has anchored the virtual dashboard 200mm above the patient's shoulder; the image processing module continuously tracks the precise pixel position of the physical landmark in each frame of the image; simultaneously, the rendering engine records the rendered pixel position of the virtual dashboard on the screen, at which point the spatial stability risk scoring model... Sets the average pixel distance (e.g., Euclidean distance) between the actual rendering position of the virtual layer and its expected rendering position calculated based on the physical anchor target, and smooths it within the time window.

[0143]

[0144] Where N is the number of frames within the time window, and u,v are the pixel coordinates. For virtual pixel coordinates, These are the desired pixel coordinates.

[0145] Scenario 2: The AR glasses wearer makes minor head adjustments during surgery, or the operating room lighting changes rapidly, causing a slight drift in the SLAM system.

[0146] In response, over a continuous 50 frames (approximately 0.5 seconds), the system detected an average change in the actual rendered position of the virtual dashboard relative to the desired position. The deviation is set to an example spatial stability risk threshold of 5 pixels. If the threshold is exceeded, the augmented reality glasses will immediately overlay a dynamically flashing blue border on the edge of the virtual dashboard, displaying the warning text "AR spatial positioning is unstable" inside. At the same time, the dashboard itself will become slightly translucent and vibrate slightly at a frequency of 0.5Hz to remind the doctor that the current AR position may be inaccurate, and that a brief system reset or manual calibration may be needed to restore high-precision positioning.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0148] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0149] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0150] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0151] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0152] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An AR-based visualization and guidance system for muscle relaxation, characterized in that, include: The muscle relaxation monitoring module is used to collect patients' muscle relaxation monitoring data in real time and output four sets of stimulation ratios; The image perception module is used to capture real-time environmental images of the surgical field and acquire pose tracking data; An image processing module, connected to the muscle relaxation monitoring module and the image perception module, is used to construct a real-time surgical space coordinate system, convert the four series of stimulus ratios into a virtual information layer, and calculate the projection parameters of the virtual information layer; An augmented reality display terminal, connected to the image processing module, is used to overlay the virtual information layer onto the physical target location within the target's field of view.

2. The system according to claim 1, characterized in that, The image processing module includes: The data preprocessing unit is used to preprocess the ratios of the four cascaded stimuli. Spatial anchoring unit is used to calculate the coordinate mapping relationship between virtual objects and the physical world based on image information and pose data; The rendering engine unit is used to generate virtual instruments or color maps with dynamic visual effects in real time based on the preprocessed values ​​of the ratios of four strings of stimuli.

3. The system according to claim 1, characterized in that, The augmented reality display terminal uses optical waveguide components to guide the micro-display signals generated by the image processing module to the human retina while maintaining the transmittance of external ambient light.

4. The system according to claim 1, characterized in that, The image processing module further includes: A visual data integrity monitoring unit is used to: receive multimodal physiological data related to the patient's physiological state, wherein the multimodal physiological data includes at least one of heart rate, blood oxygen saturation, or end-tidal carbon dioxide partial pressure; An evaluation unit is used to evaluate the consistency between the muscle relaxation state displayed by the virtual information layer and the multimodal physiological data, so as to generate a physiological consistency risk score. The monitoring unit is used to continuously monitor the visual spatial position stability of the virtual information layer relative to the physical anchoring target, so as to generate a spatial stability risk score. The overlay layer unit is used to control the augmented reality display terminal to generate and overlay an abnormal warning layer when the physiological consistency risk score or spatial stability risk score exceeds a preset threshold.

5. An AR-based visualization guidance method for muscle relaxation state, characterized in that, include: Acquire real-time muscle relaxation monitoring data of patients collected by a muscle relaxation monitoring device, wherein the real-time muscle relaxation monitoring data includes four series of stimulation ratios; Capture image information of the surgical field of view, and construct a real-time surgical space coordinate system containing physically anchored targets based on the image information; Four sets of stimulus ratios are input into a preset visual mapping model to generate a corresponding virtual information layer, wherein the virtual information layer includes color features and graphic geometric features associated with the values ​​of the four sets of stimulus ratios. Calculate the projection parameters of the virtual information layer in the augmented reality glasses display unit based on the real-time surgical space coordinate system; The augmented reality glasses are used to overlay and project the virtual information layer onto a preset position in the surgical field of view to achieve immersive monitoring of the patient's muscle relaxation state.

6. The method according to claim 5, characterized in that, The construction of a real-time surgical space coordinate system containing physically anchored targets based on the image information includes: Multi-frame video streams of the surgical field of view are obtained through the external camera of augmented reality glasses; Identify preset physical anchoring markers in the video stream, or identify surface features of the patient's surgical site; Establish the rotation and translation transformation matrix between the world coordinate system of the augmented reality glasses and the physical anchor target.

7. The method according to claim 5, characterized in that, The step of inputting the four sequential stimulus ratios into a preset visual mapping model to generate corresponding virtual information layers includes: The four cascaded stimulus ratios are mapped to a preset continuous color space; The pointer deflection angle of the virtual dashboard is determined based on the ratio of the four sets of stimuli, or the transparency parameter or flashing frequency of the virtual color block is determined.

8. The method according to claim 5, characterized in that, The method further includes: Acquire multimodal physiological data related to the patient's physiological state, wherein the multimodal physiological data includes at least one of heart rate, blood oxygen saturation, or end-tidal carbon dioxide partial pressure; The consistency between the muscle relaxation state displayed by the virtual information layer and the multimodal physiological data is evaluated to generate a physiological consistency risk score; The visual spatial position stability of the virtual information layer relative to the physical anchored target is continuously monitored to generate a spatial stability risk score. If the physiological consistency risk score or spatial stability risk score exceeds a preset threshold, an abnormality warning layer is generated and overlaid in the augmented reality glasses.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 5 to 8 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 5 to 8.

Images