Bronchoscope intraoperative navigation system based on multi-sensor fusion

Abstract

The invention discloses a bronchoscope intraoperative navigation system based on multi-sensor fusion, and relates to the technical field of medical treatment, and according to the bronchoscope intraoperative navigation system based on multi-sensor fusion, multiple sensors are integrated through a multi-modal sensing module, and all-directional accurate positioning is achieved; the data fusion control module synchronizes time signals and adopts unscented Kalman filtering, so that the data reliability is improved; the dynamic compensation module predicts bronchial deformation by means of an LSTM neural network and reduces the influence of respiratory movement; the navigation display module visually presents the path and the position of the bronchoscope body through high-resolution three-dimensional reconstruction and the AR technology, the system is efficient in working process, short in positioning delay, small in error and capable of achieving automatic repositioning, submillimeter navigation can be achieved without intraoperative CT verification, preoperative model construction is rapid, and the accuracy, safety and efficiency of bronchoscope surgery can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of medical technology, and in particular to a bronchoscopic intraoperative navigation system based on multi-sensor fusion. Background Art

[0002] The existing bronchoscopic intraoperative navigation system based on multi-sensor fusion still has the following drawbacks in actual use:

[0003] Currently, bronchoscopic surgery mainly relies on preoperative CT images and electromagnetic navigation (ENB). However, intraoperative "image drift" is easily caused by respiratory movement and tissue deformation, and navigation accuracy decreases by 30%-50%. Among them, the system lacks the ability to temporally and spatially align real-time multi-sensor (such as optical, electromagnetic, and ultrasonic) data, and different sampling frequencies (electromagnetic sensor 100Hz vs. optical 30fps) result in fusion lags of >200ms. Traditional algorithms do not establish a respiratory motion model, and the positioning error in the lower lobe area (respiratory displacement >15mm) can reach 3-5mm, which cannot meet the needs of submillimeter biopsy. In addition, the existing system requires manual marking of anatomical feature points, which adds 15-20 minutes to preoperative preparation time, and frequent CT verification is required during surgery, which increases the patient's radiation dose by 35%. Summary of the Invention

[0004] The purpose of the present invention is to provide a bronchoscopic intraoperative navigation system based on multi-sensor fusion to solve the above-mentioned problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a bronchoscopic intraoperative navigation system based on multi-sensor fusion, comprising:

[0006] A multimodal sensing module, comprising an electromagnetic positioning unit embedded in the front end of the bronchoscope, an optical tracking unit fixed to a marker on the bronchoscope body, and an inertial measurement unit integrated into the bronchoscope body;

[0007] a data fusion control module, wherein the data fusion control module synchronizes the clock signals of the electromagnetic positioning unit, the optical tracking unit, and the inertial measurement unit through the IEEE 1588 precision time protocol, with a synchronization error of less than 1 ms;

[0008] A dynamic compensation module, which receives real-time data from the multimodal perception module and outputs posture correction instructions;

[0009] A navigation display module, wherein the navigation display module displays the virtual path and the real-time scope position in an overlay manner;

[0010] The data fusion control module and the dynamic compensation module interact via the PCIe 3.0 bus, with a transmission delay of less than 0.5 ms.

[0011] Furthermore, the electromagnetic positioning unit includes a three-axis Hall sensor array, the positioning accuracy of the three-axis Hall sensor array is 0.5 mm, the sampling frequency is 100 Hz, and the output signal thereof is processed by adaptive noise reduction, and the noise reduction algorithm satisfies:

[0012]

[0013] in is the original signal, is the observation matrix, is the difference operator, =0.3 is the sparse coefficient.

[0014] Furthermore, the data fusion control module executes an unscented Kalman filter algorithm, which dynamically allocates sensor weight coefficients, wherein the weight of the electromagnetic positioning unit is 0.6, the weight of the optical tracking unit is 0.3, and the weight of the inertial measurement unit is 0.1, and the state prediction equation satisfies:

[0015]

[0016] in is the Sigma point set, is the mean weight, The status dimension.

[0017] Furthermore, the dynamic compensation module includes a pre-trained LSTM neural network. The input of the LSTM neural network is the respiratory motion model derived from the preoperative CT and the real-time chest pressure data during the operation. The output of the LSTM neural network is the bronchial deformation prediction vector of the next respiratory cycle. The hidden layer activation function is:

[0018]

[0019] in is the weight matrix, is the bias term, the time step .

[0020] Furthermore, the training set of the respiratory motion model contains more than 500 patient-specific CT sequences. The displacement error predicted by the respiratory motion model is less than 0.8 mm, and its loss function adopts Huber loss:

[0021]

[0022] in is the threshold parameter.

[0023] Furthermore, the navigation display module includes:

[0024] A three-dimensional reconstruction unit, wherein the three-dimensional reconstruction unit uses a 3D U-Net to segment the bronchial tree structure;

[0025] AR overlay unit, which highlights the planned path in green and dynamically marks the real-time mirror position in red, with the deviation between the two visualized as vector arrows;

[0026] The output resolution of the 3D reconstruction unit reaches 0.3 mm³ / voxel, and the segmentation Dice coefficient is greater than 0.91.

[0027] Furthermore, the 3D reconstruction unit integrates the Harris corner detection algorithm, which can automatically identify bronchial bifurcation points in sub-second time with an accuracy rate greater than 92%. Its corner response function is:

[0028]

[0029] in is the second-order moment matrix of the image, is an empirical constant.

[0030] Furthermore, the workflow of the system includes:

[0031] S1: The electromagnetic positioning unit and the optical tracking unit synchronously collect the spatial coordinates of the mirror body;

[0032] S2: The inertial measurement unit compensates for rapid motion artifacts of the bronchoscope;

[0033] S3: The data fusion control module outputs time-aligned fusion pose data;

[0034] S4: The dynamic compensation module generates a path correction instruction according to the fused posture data;

[0035] S5: The navigation display module updates the AR navigation interface;

[0036] S6: When it is detected that the positioning error exceeds 1.5 mm, the system automatically triggers the optical repositioning process.

[0037] Furthermore, the delay of the time-aligned fused posture data in S3 is less than 80ms, and the positioning error of the fused posture data in the deep breathing state is less than 1.2mm.

[0038] Furthermore, the system achieves submillimeter navigation without intraoperative CT verification, and the preoperative model construction time of the system is less than 3 minutes.

[0039] Compared with the existing technology, the bronchoscopic intraoperative navigation system based on multi-sensor fusion provided by the present invention has the following beneficial effects:

[0040] This bronchoscopic intraoperative navigation system based on multi-sensor fusion integrates multiple sensors through a multimodal perception module to achieve all-round precise positioning; the data fusion control module synchronizes time signals and adopts unscented Kalman filtering to improve data reliability; the dynamic compensation module uses the LSTM neural network to predict bronchial deformation and reduce the impact of respiratory movement; the navigation display module uses high-resolution three-dimensional reconstruction and AR technology to intuitively present the path and scope position. The system has an efficient workflow, short positioning delay, small error, automatic repositioning, and can achieve submillimeter navigation without intraoperative CT verification. The preoperative model is quickly constructed, which can significantly improve the accuracy, safety and efficiency of bronchoscopic surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0042] Figure 1 Schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0044] See also Figure 1 , a bronchoscopic navigation system based on multi-sensor fusion, including:

[0045] A multimodal sensing module, which includes an electromagnetic positioning unit embedded in the front end of the bronchoscope, an optical tracking unit fixed to the scope's body markers, and an inertial measurement unit integrated into the scope body;

[0046] Data fusion control module, which synchronizes the clock signals of the electromagnetic positioning unit, optical tracking unit, and inertial measurement unit through the IEEE 1588 precision time protocol, with a synchronization error of less than 1ms;

[0047] Dynamic compensation module, which receives real-time data from the multimodal perception module and outputs posture correction instructions;

[0048] A navigation display module, which displays the virtual path and the real-time mirror position in a superimposed manner;

[0049] The data fusion control module and the dynamic compensation module interact through the PCIe3.0 bus, and the transmission delay is less than 0.5ms.

[0050] The electromagnetic positioning unit includes a three-axis Hall sensor array with a positioning accuracy of 0.5mm and a sampling frequency of 100Hz. Its output signal undergoes adaptive noise reduction processing, and the noise reduction algorithm meets the following requirements:

[0051]

[0052] in is the original signal, is the observation matrix, is the difference operator, =0.3 is the sparse coefficient.

[0053] A customized electromagnetic positioning unit is embedded in the front end of the bronchoscope, with a three-axis Hall sensor array as its core. During the actual manufacturing process, micro-electromechanical systems (MEMS) technology is used to integrate the sensor array on a tiny chip to adapt to the narrow space at the front end of the bronchoscope. The positioning accuracy reaches 0.5mm, which is due to the high sensitivity design of the sensor and the precise calibration process. During the production stage, each sensor is individually calibrated and tested in a standard magnetic field environment to adjust the sensor parameters to ensure its measurement accuracy. The sampling frequency is set to 100Hz, which can quickly capture the position changes of the bronchoscope. For the output signal, an adaptive noise reduction processing algorithm is used. The original signal in the formula is obtained by sensor acquisition, and the observation matrix is pre-set according to the characteristics of the sensor and the measurement environment. The differential operator is used to extract the change characteristics of the signal. The sparse coefficient λ=0.3 is the optimal parameter obtained by optimization through a large amount of experimental data. In actual application, the collected original signal is input into the algorithm, and after computational processing, the noise interference is removed to obtain an accurate position signal.

[0054] Multiple suitable locations are selected on the bronchoscope body and fixed marking points; the marking points are made of reflective material and can efficiently reflect the light emitted by external optical equipment; the external optical tracking device is equipped with a high-resolution camera and a high-precision optical positioning algorithm; in the surgical environment, the optical tracking device captures the reflected light from the marking points in real time, and determines the position and posture of the bronchoscope body by calculating the angle and distance of the light; this process performs multiple calculations per second to achieve real-time tracking of the bronchoscope.

[0055] An inertial measurement unit (IMU) is integrated into the body of the bronchoscope. The IMU contains an accelerometer and a gyroscope. The accelerometer measures the acceleration of the bronchoscope in three dimensions, while the gyroscope measures its angular velocity. During installation, the axis of the IMU is precisely aligned with the axis of the bronchoscope to ensure the accuracy of the measurement data. During surgery, the accelerometer and gyroscope collect data in real time, convert the analog signal into a digital signal through the internal signal processing circuit, and transmit it to the subsequent data processing module.

[0056] The data fusion control module executes the unscented Kalman filter algorithm, which dynamically allocates sensor weight coefficients. The weight of the electromagnetic positioning unit is 0.6, the weight of the optical tracking unit is 0.3, and the weight of the inertial measurement unit is 0.1. Its state prediction equation satisfies:

[0057]

[0058] in is the Sigma point set, is the mean weight, The status dimension.

[0059] The data fusion control module synchronizes the clock signals of the electromagnetic positioning unit, optical tracking unit, and inertial measurement unit through the IEEE 1588 precision time protocol. When the system is set up, each sensor unit is equipped with a communication interface that supports the IEEE 1588 protocol. Before the operation begins, the time synchronization program is started, and each sensor unit communicates with the master clock source through the network and adjusts its own clock to reduce the synchronization error to less than 1ms.

[0060] For data fusion, the unscented Kalman filter algorithm is adopted; in the algorithm implementation process, according to the characteristics of each sensor and the actual application scenario, the weight of the electromagnetic positioning unit is pre-set to 0.6, the weight of the optical tracking unit is pre-set to 0.3, and the weight of the inertial measurement unit is pre-set to 0.1; when the system is running, first, a Sigma point set is generated according to the state and process noise covariance of the previous moment; then, the Sigma point set is predicted by the state transfer equation to obtain the predicted Sigma point set; then, the predicted measurement value is calculated according to the measurement model and the measurement noise covariance; then, the Kalman gain is calculated according to the actual measurement value and the predicted measurement value; finally, the state estimate of the system is updated according to the Kalman gain and the predicted state; in this process, the mean weight is determined according to the calculation result of the Sigma point set, and the state dimension is set according to the data type and quantity to be fused; through continuous iterative calculation, accurate fusion estimation of the bronchoscope position and posture is achieved.

[0061] The dynamic compensation module includes a pre-trained LSTM neural network. The input of the LSTM neural network is the respiratory motion model derived from the preoperative CT and the real-time chest pressure data during the operation. The output of the LSTM neural network is the bronchial deformation prediction vector for the next respiratory cycle. The hidden layer activation function is:

[0062]

[0063] in is the weight matrix, is the bias term, the time step .

[0064] The training set of the respiratory motion model contains more than 500 patient-specific CT sequences. The displacement error predicted by the respiratory motion model is less than 0.8 mm, and its loss function uses the Huber loss:

[0065]

[0066] in is the threshold parameter.

[0067] In the dynamic compensation module, the LSTM neural network is trained before surgery. Specific CT sequences from more than 500 patients are collected and preprocessed, including image enhancement and segmentation, to extract bronchial structural information. Simultaneously, the patient's chest pressure data during breathing is recorded and collected through sensors, followed by data cleaning and normalization. The preprocessed CT sequences and chest pressure data are used as training sets and input into the LSTM neural network for training. During training, the network parameters are adjusted to enable the network to learn the relationship between respiratory movement and bronchial deformation. The Huber loss is used as the loss function, and the threshold parameters are optimized based on the characteristics of the training data and experimental results. During surgery, the LSTM neural network receives preoperative CT-derived respiratory motion model data and intraoperative real-time chest pressure data in real time. The network calculates based on the input data, performs nonlinear transformations through the activation function of the hidden layer, and outputs the bronchial deformation prediction vector for the next respiratory cycle. The weight matrix and bias term in the activation function are determined during the training process, and the time step is set according to the frequency of data acquisition and the characteristics of the respiratory cycle. Based on the prediction vector, the dynamic compensation module generates posture correction instructions and sends them to the subsequent control unit.

[0068] The navigation display module includes:

[0069] The 3D reconstruction unit uses 3D U-Net to segment the bronchial tree structure;

[0070] AR overlay unit, the AR overlay unit highlights the planned path in green and dynamically marks the real-time mirror position in red, and the deviation between the two is visualized with a vector arrow;

[0071] The output resolution of the 3D reconstruction unit reaches 0.3 mm³ / voxel, and the segmentation Dice coefficient is greater than 0.91.

[0072] The 3D reconstruction unit integrates the Harris corner detection algorithm, which automatically identifies bronchial bifurcation points in sub-second time with an accuracy rate greater than 92%. Its corner response function is:

[0073]

[0074] in is the second-order moment matrix of the image, is an empirical constant.

[0075] The system's workflow includes:

[0076] S1: The electromagnetic positioning unit and the optical tracking unit synchronously collect the spatial coordinates of the mirror;

[0077] S2: Inertial measurement unit compensates for rapid motion artifacts of the bronchoscope;

[0078] S3: The data fusion control module outputs the time-aligned fusion pose data;

[0079] S4: The dynamic compensation module generates path correction instructions based on the fused pose data;

[0080] S5: The navigation display module updates the AR navigation interface;

[0081] S6: When the positioning error is detected to be greater than 1.5 mm, the system automatically triggers the optical repositioning process.

[0082] The delay of the time-aligned fused pose data in S3 is less than 80ms, and the positioning error of the fused pose data in deep breathing state is less than 1.2mm.

[0083] The system achieves submillimeter navigation without intraoperative CT verification, and the system's preoperative model construction time is less than 3 minutes.

[0084] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. The bronchoscopic intraoperative navigation system based on multi-sensor fusion is characterized by: include: A multimodal sensing module, comprising an electromagnetic positioning unit embedded in the front end of the bronchoscope, an optical tracking unit fixed to a marker on the bronchoscope body, and an inertial measurement unit integrated into the bronchoscope body; a data fusion control module, wherein the data fusion control module synchronizes the clock signals of the electromagnetic positioning unit, the optical tracking unit, and the inertial measurement unit through the IEEE 1588 precision time protocol, with a synchronization error of less than 1 ms; A dynamic compensation module, which receives real-time data from the multimodal perception module and outputs posture correction instructions; A navigation display module, wherein the navigation display module displays the virtual path and the real-time scope position in an overlay manner; The data fusion control module and the dynamic compensation module interact via the PCIe 3.0 bus, with a transmission delay of less than 0.5 ms.

2. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 1, characterized in that: The electromagnetic positioning unit includes a three-axis Hall sensor array. The positioning accuracy of the three-axis Hall sensor array is 0.5 mm, the sampling frequency is 100 Hz, and the output signal thereof is processed by adaptive noise reduction. The noise reduction algorithm satisfies: in is the original signal, is the observation matrix, is the difference operator, =0.3 is the sparse coefficient.

3. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 1, characterized in that: The data fusion control module executes an unscented Kalman filter algorithm, which dynamically allocates sensor weight coefficients. The weight of the electromagnetic positioning unit is 0.6, the weight of the optical tracking unit is 0.3, and the weight of the inertial measurement unit is 0.

1. The state prediction equation satisfies: in is the Sigma point set, is the mean weight, The status dimension.

4. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 1, characterized in that: The dynamic compensation module includes a pre-trained LSTM neural network. The input of the LSTM neural network is the respiratory motion model derived from the preoperative CT and the real-time chest pressure data during the operation. The output of the LSTM neural network is the bronchial deformation prediction vector of the next respiratory cycle. The hidden layer activation function is: in is the weight matrix, is the bias term, the time step .

5. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 4, characterized in that: The training set of the respiratory motion model contains more than 500 patient-specific CT sequences. The displacement error predicted by the respiratory motion model is less than 0.8 mm, and the loss function adopts Huber loss: in is the threshold parameter.

6. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 1, characterized in that: The navigation display module includes: A three-dimensional reconstruction unit, wherein the three-dimensional reconstruction unit uses a 3D U-Net to segment the bronchial tree structure; AR overlay unit, which highlights the planned path in green and dynamically marks the real-time mirror position in red, with the deviation between the two visualized as vector arrows; The output resolution of the 3D reconstruction unit reaches 0.3 mm³ / voxel, and the segmentation Dice coefficient is greater than 0.

91.

7. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 1, characterized in that: The 3D reconstruction unit integrates the Harris corner detection algorithm, which can automatically identify bronchial bifurcation points in sub-second time with an accuracy rate greater than 92%. Its corner response function is: in is the second-order moment matrix of the image, is an empirical constant.

8. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 1, characterized in that: The workflow of the system includes: S1: The electromagnetic positioning unit and the optical tracking unit synchronously collect the spatial coordinates of the mirror body; S2: The inertial measurement unit compensates for rapid motion artifacts of the bronchoscope; S3: The data fusion control module outputs time-aligned fusion pose data; S4: The dynamic compensation module generates a path correction instruction according to the fused posture data; S5: The navigation display module updates the AR navigation interface; S6: When it is detected that the positioning error exceeds 1.5 mm, the system automatically triggers the optical repositioning process.

9. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to claim 8, characterized in that: The delay of the time-aligned fused posture data in S3 is less than 80ms, and the positioning error of the fused posture data in the deep breathing state is less than 1.2mm.

10. The bronchoscopic intraoperative navigation system based on multi-sensor fusion according to any one of claims 1 to 9, characterized in that: The system achieves submillimeter navigation without intraoperative CT verification, and the preoperative model building time of the system is less than 3 minutes.

Images