Off-site mixed reality-based airway collaborative intubation safety control system for infants and young children

Abstract

The application discloses a kind of based on off-site mixed reality infant airway collaborative intubation safety control system.The system aims to solve the problem of space synchronization robustness and predictive latency compensation in remote guidance, the mapping relationship between local and remote coordinate system is established by space synchronization module, and the trajectory of remote expert is predicted and compensated by using long short-term memory network (LSTM), to eliminate visual lag.The core safety mechanism includes: airway dynamic model module generates real-time dynamic signed distance field (D-SDF);Risk prediction module calculates the dynamic collision risk probability of laryngoscope tip and dynamic airway wall in real time.Tactile feedback device responds to risk probability, adopts variable admittance control strategy, generates operation stickiness by nonlinearly increasing virtual damping, and when risk exceeds the critical value, activates the virtual stiffness term pointing to the safety area to apply reverse elastic restoring force, realize active safety avoidance in physical level, guarantee the sub-millimeter level precision and safety of infant airway operation.

Description

Technical Field

[0001] This invention belongs to the fields of computer communication, mixed reality (MR) technology, medical digital twins and telemedicine guidance, and particularly relates to a safety control system for infant airway collaborative intubation based on remote mixed reality using a 5G network. Background Technology

[0002] With the development of telemedicine technology, remote guidance using the low latency and high bandwidth characteristics of 5G has become possible. However, in infant airway intubation procedures requiring extremely high precision (such as sub-millimeter level), remote collaborative guidance faces two major technological bottlenecks:

[0003] First, because the anatomical structures of local patients and the laryngoscopes used by operators are located in the local physical coordinate system, while the guidance annotations and virtual models of remote experts are located in a remote or cloud-based virtual coordinate system, the robustness of cross-regional spatial synchronization is extremely poor. Traditional coordinate transmission is easily affected by network jitter, causing the arrows or lines drawn by experts in virtual space to fail to accurately "attach" to specific anatomical structures (such as the glottis) of local patients, resulting in guidance bias.

[0004] Secondly, although 5G technology offers extremely low latency, in practical applications, data packaging, decoding, rendering, and complex cloud computing still cause a slight "lag" in the local operator's view of remote expert gesture commands. This delay is unacceptable in minimally invasive procedures that require millisecond-level feedback, and it will greatly affect the accuracy of guidance and the smoothness of the operation.

[0005] Therefore, existing technologies lack a high-precision collaborative guidance mechanism that can overcome spatial fluctuations in different locations and achieve predictive delay compensation. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a safety control system for infant airway collaborative intubation based on remote mixed reality, characterized by comprising:

[0007] The local state perception module is configured to collect and calculate the kinematic state data of the laryngoscope tip in real time;

[0008] The airway dynamic model module is configured to update the geometric deformation of the pre-stored three-dimensional airway model based on the child's real-time physiological motion signals, and generate real-time dynamic airway wall position information.

[0009] The spatial synchronization module is configured to establish a real-time mapping relationship between the local physical coordinate system and the remote virtual coordinate system;

[0010] The remote collaboration module is configured to receive remote guidance data, use prediction algorithms to compensate for transmission delays, generate predicted guidance trajectories, and overlay them onto the local field of view.

[0011] The risk prediction module is configured to calculate the probability of dynamic collision risk based on the spatial relationship and motion trend between the kinematic state of the laryngoscope tip and the real-time dynamic airway wall position information.

[0012] The haptic feedback device is configured to adaptively adjust the mechanical resistance parameters output to the operating handle in response to the dynamic collision risk probability, so as to apply active haptic intervention when the risk increases.

[0013] Specifically, the airway dynamic model module generates real-time dynamic airway wall position information, which includes: constructing a static signed distance field based on preoperative medical images to characterize the minimum Euclidean distance from a spatial point within the airway to the surface of the airway wall, wherein the positive or negative value in the static signed distance field is used to distinguish between the safe area inside the airway and the wall penetration area; calculating the real-time respiratory phase based on real-time collected physiological motion signals using a respiratory motion function, and calculating the deformation field vector corresponding to the respiratory phase based on a preset statistical shape model; using the deformation field vector to perform reverse offset query and correction on the static signed distance field to generate a dynamic signed distance field containing the time dimension; outputting the dynamic airway wall position information in real time based on the dynamic signed distance field, and determining the real-time normal vector direction of the airway wall surface by calculating the gradient direction of the dynamic signed distance field.

[0014] Specifically, the prediction algorithm in the remote collaboration module employs a Long Short-Term Memory (LSTM) network, configured as follows: the input layer receives the remote expert gesture sequence within a historical time window; the hidden layer uses a gating mechanism to extract the temporal motion features of the expert's operation; and the output layer predicts the gesture position at a future time step to compensate for the visual lag caused by network transmission.

[0015] Specifically, in the risk prediction module, calculating the dynamic collision risk probability includes: calculating the projection of the velocity vector at the end of the laryngoscope onto the real-time normal vector direction of the dynamic airway wall surface to obtain the relative approach speed; constructing a nonlinear probability density function, outputting an exponentially increasing risk probability value when the relative approach speed points towards the airway wall and the distance is less than a safety threshold; and introducing an attenuation factor to suppress the risk probability value when the relative approach speed moves away from the airway wall.

[0016] Specifically, the haptic feedback device employs a variable admittance control strategy to adjust the mechanical resistance parameters, including: the mechanical resistance parameters comprising a virtual damping coefficient and a virtual stiffness coefficient; the haptic feedback device is configured to apply a reverse elastic restoring force pointing towards a safe area at the physical level by adjusting the virtual stiffness coefficient, thereby constructing a no-entry field to prevent entry into the dangerous area; wherein, when the probability of dynamic collision risk increases, the virtual damping coefficient is nonlinearly increased to generate a sense of operational viscosity; when the probability of dynamic collision risk exceeds a preset critical value, the virtual stiffness term pointing towards the safe area is activated to generate a reverse elastic restoring force.

[0017] Specifically, the local state perception module uses the Error State Kalman Filter (ESKF) algorithm to calculate the kinematic state data of the laryngoscope tip, which includes: dividing the kinematic state into a nominal state and an error state; performing integral prediction of the nominal state of the laryngoscope tip and observation correction of the error state by fusing high-frequency inertial measurement data and low-frequency position tracking data, and outputting kinematic state data including position uncertainty; the position uncertainty is input as a parameter to the risk prediction module to adjust the sensitivity of risk calculation.

[0018] Specifically, the method for establishing a real-time mapping relationship by the spatial synchronization module includes: using computer vision algorithms to identify anatomical semantic anchor points in the endoscopic video stream; based on the anatomical semantic anchor points, constructing a weighted registration objective function including topological rigidity constraints, and solving for the rigid transformation matrix between the local and remote coordinate systems.

[0019] Specifically, the anatomical semantic anchor points are reference points with spatial coordinate attributes extracted from key anatomical structures in the airway identified in the endoscopic video stream, including the anterior commissure of the glottis and / or the apex of the arytenoid cartilage and / or the carina.

[0020] Specifically, the topological rigidity constraint term is configured to calculate the pre-existing relative geometric distances or angular relationships of the key anatomical structures within the airway in the three-dimensional model space, and penalize any rigid transformation matrix solution results that cause significant deviations from the pre-existing relative geometric distances or angular relationships, in order to ensure the anatomical accuracy and robustness of spatial synchronization.

[0021] Specifically, when the probability of dynamic collision risk exceeds a preset critical value, a reverse elastic restoring force is generated pointing towards the safe area. This includes: constructing a no-entry field, which is activated when the minimum distance is less than a preset safety margin distance; the reverse elastic restoring force is consistent with the normal vector direction of the dynamic airway wall surface, and the normal vector direction points towards the safe area at the center of the airway, so as to physically limit the displacement of the laryngoscope tip in the dangerous direction.

[0022] The present invention proposes a safety control system for infant airway coordination intubation based on remote mixed reality, which has the following significant advantages compared with traditional video navigation and passive assistance systems:

[0023] 1. Achieved sub-millimeter level high-precision motion sensing, effectively overcoming positioning interference in narrow spaces.

[0024] By employing the Error State Kalman Filter (ESKF) algorithm in the local state perception module, this invention deeply fuses high-frequency inertial measurement data (1000Hz) with low-frequency absolute position tracking data (60Hz). This dual-layer state estimation architecture effectively solves the problems of line-of-sight obstruction, electromagnetic drift, and integral accumulation errors that are prone to occur with a single sensor in the extremely narrow and complex physical environment of an infant's airway. This module can output smooth sub-millimeter pose data in real time, providing a reliable kinematic reference for subsequent high-precision hazard avoidance.

[0025] 2. It possesses real-time respiratory motion compensation capabilities, resolving modeling lag issues in non-rigid deformation environments.

[0026] The airway dynamic model module constructs a dynamic signed distance field (D-SDF) with a time dimension and introduces an inverse offset correction algorithm based on a statistical shape model. Compared to traditional frame-by-frame 3D mesh reconstruction, this invention can capture the tissue expansion and contraction caused by the child's breathing in real time without requiring high computing power. This ensures real-time alignment of the model's inner wall position with the child's anatomical structure, achieving millisecond-level collision detection response and meeting the extremely high real-time requirements of surgical operations.

[0027] 3. Significantly improved the robustness of remote spatial synchronization, eliminating visual offset caused by network fluctuations.

[0028] When establishing mapping relationships, the spatial synchronization module introduces a weighted registration algorithm that includes anatomical semantic anchor points (such as the anterior commissure of the glottis) and topological rigidity constraints. This mechanism uses the geometric distances and angles of the inherent anatomical structures of the human body as regularization constraints, which can penalize and correct coordinate mapping jumps caused by network data jitter. This ensures that the guidance annotations from remote experts are always accurately and stably "attached" to the specific location of the child's airway, avoiding medical misjudgments caused by spatial mismatch.

[0029] 4. Effectively compensates for remote collaborative transmission latency, eliminating the "lag" in operational instructions.

[0030] The remote collaboration module utilizes a Long Short-Term Memory (LSTM) network to extract the temporal features of expert operations, enabling prediction of gesture trajectories at future time steps. Combined with the high bandwidth of 5G networks, this invention can interpolate and render a "shadow guide" in real-time within the local field of vision, successfully offsetting the visual lag caused by data packaging and transmission, allowing the local operator to experience near-real-time, zero-latency guidance.

[0031] 5. It achieves a paradigm shift from "passive assistance" to "active risk avoidance," providing physical-level protection for vulnerable organizations.

[0032] The risk prediction module, through relative velocity projection logic, can accurately distinguish between two motion trends: "stationary approach" and "rapid collision," and dynamically adjusts the risk sensitivity based on positional uncertainty. Combined with the variable admittance control strategy employed by the tactile feedback device, the system can generate a "viscous feeling" by increasing virtual damping when the risk increases, forcing the operator to slow down; when approaching the critical threshold, it activates a virtual stiffness term (soft lock) pointing towards a safe area, physically restricting the directional movement of the laryngoscope. This mechanism achieves "zero-distance" physical protection for vulnerable tissues such as the subglottic stenosis, significantly reducing the risk of mechanical damage during airway surgery in infants and young children.

[0033] 6. It reduces the psychological burden and technical threshold for operators, and improves the overall efficiency of collaborative surgery.

[0034] According to usability assessments, the multimodal (visual-tactile) intervention mechanism of this invention significantly reduced operators' NASA-TLX quantitative psychological load scores. Through automated risk warning and physical guidance, even inexperienced physicians can maintain consistency in their operational trajectory under the precise guidance of remote experts, thereby effectively shortening the completion time of intubation tasks and improving the radiation efficiency of national-level high-quality medical resources. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the infant airway collaborative intubation safety control system based on remote mixed reality proposed in this invention. Detailed Implementation

[0036] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] This invention proposes a safety control system for infant airway co-intubation based on remote mixed reality, comprising:

[0038] The local state perception module is configured to collect and calculate the kinematic state data of the laryngoscope tip in real time;

[0039] The local state perception module employs the Error State Kalman Filter (ESKF) algorithm, which divides the kinematic state into "nominal state" and "error state." By fusing high-frequency inertial measurement data and low-frequency position tracking data, the nominal state at the end of the laryngoscope is integrally predicted and the error state is observed and corrected, outputting kinematic state data including position uncertainty. The position uncertainty is input as a parameter to the risk prediction module to adjust the sensitivity of risk calculation.

[0040] This module addresses the issue of single sensors being susceptible to obstruction and drift within the confined space of the airway. It employs an Error-State Kalman Filter (ESKF) algorithm, fusing IMU (1000Hz) and electromagnetic / optical tracking data (60Hz).

[0041] Algorithm configuration: Define the system's nominal state variables and error state variables Nominal state ,in These represent the position, velocity, and quaternion orientation of the laryngoscope tip. Indicates time, It has zero bias. This indicates the deviation of the accelerometer in the inertial measurement unit (IMU). Indicates time The deviation of the gyroscope in the inertial measurement unit (IMU), Indicates accelerometer, Represents angular velocity.

[0042] Step S1.1: IMU Integral Prediction (ProcessUpdate) Based on the acceleration measured by the high-frequency IMU and angular velocity Update the nominal state:

[0043]

[0044]

[0045] in, This represents the coordinates of the laryngoscope tip at the current time t. This indicates that the tip of the laryngoscope was at the previous moment. Location coordinates, This indicates that the tip of the laryngoscope was at the previous time t. The velocity vector of 1 Indicates the time interval between two consecutive occurrences. It is a 3×3 rotation matrix. Represents the quaternion pose based on the previous time step. The resulting rotation matrix is ​​used to transform the acceleration measurement values ​​in the sensor body coordinate system to the global coordinate system; The accelerometer in the inertial measurement unit (IMU) The time deviation is one of the state variables estimated by ESKF. This is the acceleration due to gravity.

[0046] Simultaneously, calculate the error state covariance matrix. Prior estimates: .

[0047] in, Let represent the prior covariance matrix, and let represent the prior covariance matrix at the previous time step. At that time, the system's error state variable The estimation error, The state transition Jacobian matrix is ​​used to linearize the nonlinear state transition function, since laryngoscope motion (such as position, velocity, and attitude) is a nonlinear system.

[0048] Let the prior covariance matrix be the value at the current time step. The system controls the error state variable at the end of the laryngoscope ( The prediction uncertainty. Represents the process noise covariance matrix. This represents the Jacobian matrix of the noise input.

[0049] Step S1.2: Measurement Update (Measurement Correction) When low-frequency location data Upon arrival, calculate the Kalman gain. And update the error status. The error is then injected into the nominal state, and the error state is reset.

[0050]

[0051] Where K is a weight matrix, called the Kalman gain, used to determine how much trust the system should give to the predicted value model and how much trust should be given to the actual measured values ​​when fusing sensor measurements. The Kalman gain aims to minimize the covariance of the fused state estimation error, while the V matrix directly reflects the sensor's quality and noise level. This represents the observed Jacobian matrix.

[0052] Output: Real-time output of smooth kinematic states and location uncertainty variance .Should It will be input as a sensitivity parameter into the risk prediction module. Representation matrix The trace is the sum of its main diagonal elements. For a matrix, its diagonal elements are respectively The variance of the position estimation error in the direction. If the position error vector at the end of the laryngoscope... Having in three-dimensional space The three components have the following variances: , , ,but The calculation formula is:

[0053]

[0054] The airway dynamic model module is configured to update the geometric deformation of a pre-stored 3D airway model based on the child's real-time physiological motion signals, generating real-time dynamic airway wall position information. The method for generating this information includes: constructing a static signed distance field (SDF) to characterize the minimum Euclidean distance from a spatial point to the airway inner wall surface; constructing a static signed distance field based on preoperative medical images to characterize the minimum Euclidean distance from a spatial point within the airway to the airway inner wall surface, wherein the positive and negative values ​​in the static signed distance field are used to distinguish between the safe area inside the airway and the wall penetration area; introducing a respiratory motion function and calculating a deformation field vector based on the real-time respiratory phase; and using the deformation field vector to perform a reverse offset correction on the static signed distance field to generate a dynamic signed distance field (D-SDF) including a time dimension.

[0055] To address the non-rigid airway deformation caused by the child's breathing, this module generates a timestamped dynamic signed distance field (DynamicSDF). Its algorithm configuration includes:

[0056] Static field construction: Constructing a static distance field based on preoperative CT scans The numerical value represents a point in space. Euclidean distance to the nearest airway wall This indicates that the area is inside the airway, i.e., in a safe state. This indicates penetration, meaning a collision occurs.

[0057] Respiratory manifold driven: Introducing respiratory motion functions Real-time acquisition of respiratory phase The input is physical time. The output is an angle value within the interval [0, 2π]. For example: =0 indicates the start of inhalation. =π represents the end of inhalation / beginning of exhalation. =2π represents the end of expiration. This addresses the issue of uneven respiratory rate in children. Regardless of the respiratory rate, as long as the phase is the same, we assume that the airway geometry is similar. It is a respiratory motion function; input: respiratory phase. Output: Three-dimensional deformation field (or displacement vector) of the airway. It is a spatial displacement field: it is a three-dimensional spatial function. Its function is based on... The output parameters are specific points in the computation space. The displacement vector relative to the static position describes "the point at the current breathing moment". "Where exactly did it move to?"

[0058] Inverse deformation field calculation: Calculate the deformation displacement field using a pre-trained statistical shape model (SSM). Real-time dynamic distance information Obtained through reverse lookup:

[0059]

[0060] This method avoids the high computational cost of reconstructing a 3D mesh every frame and achieves millisecond-level collision detection.

[0061] The spatial synchronization module is configured to establish a real-time mapping relationship between the local physical coordinate system and the remote virtual coordinate system; this module is responsible for the rigid registration between the local anatomical space and the remote virtual space. Anatomical semantic anchor points in the endoscopic video stream are identified using computer vision algorithms; based on these semantic anchor points, a weighted registration objective function including topological rigidity constraints is constructed, and the rigid transformation matrix between the local and remote coordinate systems is calculated.

[0062] The algorithm configuration specifically includes: Semantic anchor point extraction: running a lightweight U-Net network to process the endoscopic video stream and identify key anatomical structures (such as the anterior commissure of the glottis). Apex of the arytenoid cartilage Output 2D pixel coordinate set and confidence level .

[0063] Topological constraint PnP solution: Construct a weighted PnP objective function to solve for camera pose. Introducing topological constraint terms To punish incorrect matches that violate anatomical structures:

[0064]

[0065] in These are the corresponding 3D points in the airway model. This is the projection function.

[0066] Represents the robust loss function. Indicates the observed 2D pixel coordinates. This represents the intrinsic parameter matrix of the camera. Represents the observed 2D points With 3D model points After R, , The squared distance between 2D points obtained by reprojection after transformation This represents the constraint weighting factor, used to balance reprojection errors and topological rigidity constraints. The importance coefficient; R is a 3×3 rotation matrix used to describe the attitude difference between the remote expert virtual coordinate system and the local physical coordinate system (i.e., the laryngoscope endoscope camera). It is a 3×1 vector used to describe the positional difference between the remote expert virtual coordinate system and the local physical coordinate system (i.e., the laryngoscope endoscope camera).

[0067] Topological rigid constraint terms As a regularization term in the overall optimization objective function, it is used to penalize any registration solution that violates the pre-existing anatomical geometric relationships. The complete calculation formula consists of the distance constraint term. and angle constraint terms Composed of:

[0068]

[0069] Among them, distance constraint term The calculation is as follows:

[0070]

[0071] in, This is for this pair of anchor points. The penalty weight applied to the rigid distance deviation between them. This represents a 3×3 rotation matrix. This represents the original three-dimensional coordinate vector of the anatomical semantic anchor point in the model. This represents the rigid distance between anchor points that is pre-stored in the model.

[0072] And, angle constraint terms The calculation is as follows:

[0073]

[0074] Represents anchor triplet Pre-stored inherent angles in the model, Indicates the result after the current transformation Post-anchor triplet The resulting real-time angle value; This represents the weight coefficient corresponding to each item. Let each represent a set of anchor points participating in distance constraints and angle constraints, respectively.

[0075] Each represents three distinct semantic anchor points pre-identified and defined in the airway 3D model space. These anchor points are typically key anatomical structures with stable relative positions that are identifiable in the endoscopic video stream, such as: This refers to the anterior commissure of the glottis. 'k' represents the apex of the arytenoid cartilage; 'k' represents an optional third key point, such as the carina tracheae. The semantic anchors are reference points with spatial coordinate attributes extracted from key anatomical structures within the airway (such as the anterior commissure of the glottis).

[0076] The remote collaboration module is configured to receive remote guidance data, compensate for transmission latency using a prediction algorithm, generate a predicted guidance trajectory, and overlay it onto the local field of view. The prediction algorithm in the remote collaboration module employs a Long Short-Term Memory (LSTM) network, configured as follows: the input layer receives a sequence of remote expert gestures within a historical time window; the hidden layer extracts the temporal motion features of the expert's actions using a gating mechanism; and the output layer predicts the gesture position at a future time step to compensate for visual lag caused by network transmission. To address potential latency and jitter in 5G / 6G networks, LSTM is used for trajectory prediction compensation.

[0077] Algorithm configuration: Input is past data. Expert gesture sequence of frames LSTM cells utilize gating mechanisms to extract temporal features:

[0078]

[0079] System output future Predicted location at time And render the "shadow guide" in the local MR field of view to eliminate visual lag.

[0080] The risk prediction module is configured to calculate the probability of dynamic collision risk based on the spatial relationship and motion trend between the kinematic state of the laryngoscope tip and the real-time dynamic airway wall position information.

[0081] The logic of the risk prediction module for calculating the dynamic collision risk probability includes: calculating the projection of the velocity vector at the end of the laryngoscope onto the normal vector on the surface of the dynamic airway wall to obtain the relative approach speed; constructing a nonlinear probability density function, outputting an exponentially increasing risk probability value when the relative approach speed points towards the airway wall and the distance is less than a safety threshold; and introducing an attenuation factor to suppress the risk probability value when the relative approach speed moves away from the airway wall.

[0082] This module calculates the dynamic collision probability (DCP).

[0083] The algorithm configuration includes relative velocity projection: calculating the velocity at the tip of the laryngoscope. In the dynamic airway wall normal vector The projection on the airway wall, minus the respiratory velocity of the airway wall itself. :

[0084]

[0085] Indicates the end of the laryngoscope Along the normal direction of the airway wall The velocity component of directional motion. This value is a key indicator distinguishing between "static approach" and "rapid collision," and is used to drive the nonlinear risk probability function. This represents the velocity vector at the tip of the laryngoscope, which is the real-time three-dimensional velocity vector at the tip of the laryngoscope. It represents the respiratory velocity of the airway wall and is the velocity of the closest point on the airway wall surface. The child's real-time velocity vector. This velocity is calculated by the airway dynamics model module based on the child's real-time respiratory phase and describes the expansion or contraction caused by respiration. This indicates the position of the laryngoscope tip, which is the real-time position of the laryngoscope tip in a three-dimensional coordinate system; The gradient of the distance field is represented by the position at the end of the laryngoscope. At this location, the gradient direction of the signed distance field (SDF) Φ. This gradient direction mathematically defines precisely the normal to the airway wall surface.

[0086] Nonlinear probabilistic models: combining distance Relative velocity and uncertainty Construct risk function :

[0087]

[0088] When the laryngoscope is rapidly approached to the airway wall This significantly increases the risk value; when evacuating, a decay factor is introduced. To prevent false alarms; among them, The distance sensitivity coefficient is the coefficient applied to the minimum real-time distance. The weighting factor is used to adjust the rate at which the risk probability increases as the spatial gap decreases. The speed sensitivity coefficient is applied to relatively close speeds. The weighting factor is used to adjust the contribution of dynamic trends to risk prediction.

[0089] The haptic feedback device is configured to adaptively adjust the mechanical resistance parameters output to the operating handle in response to the dynamic collision risk probability, so as to apply active haptic intervention when the risk increases.

[0090] The tactile feedback device employs a variable admittance control strategy to adjust the mechanical resistance parameters: the mechanical resistance parameters include a virtual damping coefficient and a virtual stiffness coefficient; when the probability of dynamic collision risk increases, the virtual damping coefficient is nonlinearly increased to generate a sense of operational stickiness; when the probability of dynamic collision risk exceeds a preset critical value, the virtual stiffness term pointing to the safe area is activated to generate a reverse elastic restoring force.

[0091] Based on the calculation The variable admittance control strategy is implemented to adjust the mechanical resistance of the handle.

[0092] Control law design: Setting virtual dynamic equations:

[0093]

[0094] This represents the virtual inertia term, maintaining a constant feel and simulating the inertia of a lightweight gamepad; where... The virtual inertia matrix represents the system. This represents a virtual damping term used to impede the laryngoscope's movement along the velocity direction. The movement produces a "sticky" feeling. It is a virtual damping coefficient that dynamically adjusts with risk; It is a virtual stiffness term, used to generate a restoring force when the laryngoscope deviates from the safe path, pushing the laryngoscope back to the safe area; It is a virtual stiffness coefficient that dynamically adjusts with risk. It represents the centerline of the airway or the location of a safety target. The user inputs force; the operator applies the actual physical force through the handle, which is used as input to the system to drive the laryngoscope's movement; ultimately, the desired trajectory of the laryngoscope's tip is calculated. Adaptive damping: used to create a "viscous" feeling.

[0095]

[0096] , where is the weighting coefficient, representing a moderating factor for the contribution of balancing risk to the increase in damping, used to adjust the system's sensitivity to risk according to clinical needs. Indicates basic damping. This represents the identity matrix, used to achieve isotropic damping, meaning that damping is applied consistently in all directions.

[0097] Virtual stiffness: Used to generate "reverse thrust". Only when... Activated at (e.g., 0.9):

[0098]

[0099] This force "pushes" the laryngoscope back to a safe area, achieving proactive safety avoidance. Indicates the stiffness amplification factor. This represents a non-linear activator, used to achieve proactive intervention through mutations, avoiding unnecessary interference at low risk. This indicates the preset critical threshold. This represents the dynamic airway wall normal vector, a unit normal vector perpendicular to the airway wall surface at the end of the laryngoscope. It determines the direction of the reverse thrust, ensuring the torque points towards a safe area. This stiffness term provides directional drag (Virtual Stiffness). When When the critical hazard level (such as Level 3) is reached, the system activates the virtual wall spring force. , generate with normal vector The opposing thrust in the same direction physically limits the laryngoscope's further displacement in the dangerous direction, allowing it only to be withdrawn, thus achieving active avoidance and zero-distance prevention of injury.

[0100] Virtual wall spring force This is a piecewise function used to construct a ForbiddenRegion, which is activated only when the tip of the laryngoscope enters within a preset safety margin distance. Its mathematical expression is:

[0101]

[0102] in, The virtual wall spring force vector (active safety intervention force) is represented by a three-dimensional spatial vector. This represents the stiffness coefficient of the virtual wall, which determines the strength of the reverse thrust, and is a positive scalar. : Preset safety margin distance, a positive scalar (unit: mm); : The real-time minimum distance from the tip of the laryngoscope to the dynamic airway wall, which is a positive scalar (unit: mm). The depth to which the laryngoscope tip is inserted into the safety margin area, i.e., the amount of compression of the virtual spring; The unit normal vector representing the airway wall surface is expressed as: ,in, This is the position of the end of the laryngoscope. The time; the direction points to the safe zone; Zero vector indicates that when the tip of the laryngoscope is outside the safety margin, there is no active safety intervention force.

[0103] It should be understood that the processor in the embodiments of the present invention may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0104] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0105] It should be understood that the above-described memory is exemplary but not restrictive. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0106] This application also provides a computer-readable storage medium for storing computer programs.

[0107] Optionally, the computer-readable storage medium can be applied to the terminal device in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the mobile terminal / terminal device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0108] This application also provides a computer program product, including computer program instructions.

[0109] Optionally, the computer program product can be applied to the terminal device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the mobile terminal / terminal device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0110] This application also provides a computer program.

[0111] Optionally, the computer program can be applied to the vehicle autonomous driving device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the terminal device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0112] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0113] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A safety control system for infant airway co-intubation based on remote mixed reality, characterized in that, include: The local state perception module is configured to collect and calculate the kinematic state data of the laryngoscope tip in real time; The airway dynamic model module is configured to update the geometric deformation of a pre-stored 3D airway model based on the child's real-time physiological motion signals, generating real-time dynamic airway wall position information. Specifically, this includes: constructing a static signed distance field based on preoperative medical images to characterize the minimum Euclidean distance from a spatial point within the airway to the surface of the airway wall; the positive and negative values ​​in the static signed distance field are used to distinguish between the safe area inside the airway and the wall penetration area; calculating the real-time respiratory phase based on the real-time collected physiological motion signals using a respiratory motion function, and calculating the deformation field vector corresponding to the respiratory phase based on a preset statistical shape model; using the deformation field vector to perform a reverse offset query and correction on the static signed distance field to generate a dynamic signed distance field containing a time dimension; outputting the dynamic airway wall position information in real-time based on the dynamic signed distance field, and determining the real-time normal vector direction of the dynamic airway wall surface by calculating the gradient direction of the dynamic signed distance field. The spatial synchronization module is configured to establish a real-time mapping relationship between a local physical coordinate system and a remote virtual coordinate system; the local physical coordinate system is a coordinate system based on the laryngoscope endoscope camera; the remote virtual coordinate system refers to the coordinate system of the remote expert's guidance annotations and the virtual model. The remote collaboration module is configured to receive remote guidance data, use prediction algorithms to compensate for transmission delays, generate predicted guidance trajectories, and overlay them onto the local field of view. The risk prediction module is configured to calculate the dynamic collision risk probability based on the spatial relationship and motion trend between the kinematic state of the laryngoscope tip and the real-time dynamic airway wall position information. Specifically, it includes: calculating the projection of the velocity vector of the laryngoscope tip onto the real-time normal vector direction of the dynamic airway wall surface to obtain the relative approach velocity; constructing a nonlinear probability density function, outputting an exponentially increasing risk probability value when the relative approach velocity points towards the airway wall and the distance is less than a safety threshold; and introducing an attenuation factor to suppress the risk probability value when the relative approach velocity moves away from the airway wall. A haptic feedback device is configured to adaptively adjust the mechanical resistance parameters output to the operating handle in response to the dynamic collision risk probability, so as to apply active haptic intervention when the risk increases. The haptic feedback device uses a variable admittance control strategy to adjust the mechanical resistance parameters, specifically including: the mechanical resistance parameters include a virtual damping coefficient and a virtual stiffness coefficient; the haptic feedback device is configured to apply a reverse elastic restoring force pointing towards a safe area at the physical level by adjusting the virtual stiffness coefficient, so as to construct a no-entry field that prevents entry into the danger zone; wherein, when the dynamic collision risk probability increases, the virtual damping coefficient is nonlinearly increased to generate an operating viscosity sensation; when the dynamic collision risk probability exceeds a preset critical value, a reverse elastic restoring force pointing towards a safe area is generated.

2. The system according to claim 1, characterized in that, The prediction algorithm in the remote collaboration module uses a Long Short-Term Memory (LSTM) network, which is configured as follows: the input layer receives the remote expert gesture sequence within a historical time window; the hidden layer uses a gating mechanism to extract the temporal motion features of the expert operation. The output layer predicts the gesture position at future time steps to compensate for visual lag caused by network transmission.

3. The system according to claim 1, characterized in that, The local state perception module uses the Error State Kalman Filter (ESKF) algorithm to calculate the kinematic state data of the laryngoscope tip. Specifically, it includes: dividing the kinematic state into a nominal state and an error state; integrating and predicting the nominal state of the laryngoscope tip and correcting the error state by fusing high-frequency inertial measurement data and low-frequency position tracking data, and outputting kinematic state data including position uncertainty; the position uncertainty is input as a parameter into the risk prediction module to adjust the sensitivity of risk calculation.

4. The system according to claim 1, characterized in that, The method for establishing a real-time mapping relationship by the spatial synchronization module includes: using computer vision algorithms to identify anatomical semantic anchor points in the endoscopic video stream; based on the anatomical semantic anchor points, constructing a weighted registration objective function including topological rigidity constraints, and solving the rigid transformation matrix between the local and remote coordinate systems.

5. The system according to claim 4, characterized in that, The anatomical semantic anchor points are reference points with spatial coordinate attributes extracted from key anatomical structures in the airway identified in the endoscopic video stream, including the anterior commissure of the glottis and / or the apex of the arytenoid cartilage and / or the carina.

6. The system according to claim 5, characterized in that, The topological rigidity constraint is configured to calculate the pre-existing relative geometric distances or angular relationships of key anatomical structures within the airway in the three-dimensional model space, and penalize any rigid transformation matrix solution results that cause significant deviations from the pre-existing relative geometric distances or angular relationships, in order to ensure the anatomical accuracy and robustness of spatial synchronization.

7. The system according to claim 1, characterized in that, When the probability of dynamic collision risk exceeds a preset threshold, a reverse elastic restoring force is generated pointing towards the safe area. Specifically, this includes: constructing a no-entry field, which is activated when the minimum distance is less than a preset safety margin distance; the reverse elastic restoring force is consistent with the normal vector direction of the dynamic airway wall surface, and the normal vector direction points towards the safe area at the center of the airway, so as to physically limit the displacement of the laryngoscope tip in the dangerous direction.

Images