Medical equipment automated control and positioning system based on multimodal data

Abstract

The present invention discloses an automated control and positioning system for medical equipment based on multimodal data, which relates to the technical field of medical equipment control. The system collects preoperative three-dimensional images and intraoperative ultrasound, physiological signals, equipment calibration and other multimodal data and aligns them in time; quantifies the confidence of ultrasound and physiological signals through artifact characteristics and signal-to-noise ratio; combines a displacement prediction model with the actual ultrasound displacement value, and obtains the final predicted displacement through weighted fusion according to the confidence; compares the calculated deviation of the real-time displacement of the equipment, and if it exceeds the threshold, dynamically adjusts the confidence and fusion weight based on the deviation change rate to generate an adjustment instruction; improves the anti-interference ability in complex surgical fields through multi-source data complementarity; dynamically adjusts the weight to adapt to changes in intraoperative data quality and avoids the rigidity of fixed parameters; and a closed-loop mechanism accelerates deviation convergence, thereby improving positioning accuracy compared with traditional methods. The system is suitable for surgical scenarios requiring dynamic organ compensation, such as liver cancer resection.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment control, and in particular to a medical equipment automatic control and positioning system based on multimodal data. Background Art

[0002] In minimally invasive procedures such as tumor ablation and cardiac catheterization, dynamic intraoperative organ movement (such as target drift caused by breathing and heartbeat) is a key factor affecting positioning accuracy. Clinical data show that respiratory motion can cause liver target displacement by 20-50mm and heart displacement by more than 10mm. Traditional static positioning models are unable to adapt to these dynamic changes, resulting in increased device positioning errors and significantly increased surgical risks.

[0003] Traditional medical device positioning methods mainly rely on single-modality data or multimodal fusion technology with fixed parameters, but they have significant limitations in actual clinical applications:

[0004] For example, while ultrasound imaging can provide real-time anatomical information, it is susceptible to artifacts (such as acoustic shadowing caused by blood vessels and calcifications). When the artifact area exceeds 30%, the accuracy of lesion identification in ultrasound images can drop by over 50%. Fluctuations in the reliability of single-modality data directly lead to localization errors, making it impossible to meet the high-precision requirements required during surgery. Some improved methods attempt to enhance localization robustness through multimodal data fusion (such as combining ultrasound images with physiological signals), but these methods often use fixed weights to weight displacement values. However, the intraoperative environment is dynamic: ultrasound artifacts can suddenly increase with changes in probe angle, and the signal-to-noise ratio of physiological signals can drop sharply with adjustments in patient position. Fixed weights cannot be dynamically adjusted based on real-time data quality (such as artifact area and signal-to-noise ratio), often leading to the problem of "low-confidence data dominating the fusion results." For example, even when ultrasound artifacts are severe, they are still given a high weight in the calculation, amplifying localization errors. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a medical equipment automatic control positioning method based on multimodal data.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] The medical equipment automatic control and positioning system based on multimodal data runs the medical equipment automatic control and positioning method based on multimodal data when in operation, and the method includes the following steps:

[0008] Acquiring preoperative pre-positioning data and intraoperative ultrasound images, physiological signal data, and equipment calibration data; the preoperative pre-positioning data includes a three-dimensional imaging model of the patient;

[0009] adding a unified timestamp to the intraoperative ultrasound image, physiological signal data, and device calibration data to generate multimodal data with time series alignment;

[0010] Calculating ultrasound confidence and physiological signal confidence based on artifact characteristics of the intraoperative ultrasound image and a signal-to-noise ratio of physiological signal data;

[0011] Using the time-aligned physiological signal data, the displacement prediction model generates the predicted value of the organ's dynamic displacement, and the actual displacement value of the organ is extracted based on the intraoperative ultrasound image;

[0012] According to the ultrasound confidence level and the physiological signal confidence level, weighted fusion is performed on the actual displacement value of the ultrasound image and the dynamic displacement prediction value to obtain a final predicted displacement;

[0013] The real-time displacement information of the device is extracted using the time-aligned device calibration data to obtain the deviation value between the final predicted displacement and the real-time displacement information of the device;

[0014] Determine whether the deviation value exceeds the preset deviation threshold. If so, perform the following operations:

[0015] Adjusting ultrasound confidence and physiological signal confidence based on the rate of change of the deviation value;

[0016] According to the adjusted ultrasound confidence level and physiological signal confidence level, the predicted displacement value is re-weighted and fused, and an adjustment instruction for the medical device is generated according to the predicted displacement value.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The complementary and collaborative use of multimodal data solves the problem of single-modal data easily leading to positioning failure due to data quality fluctuations. Ultrasound images provide real-time anatomical details, and physiological signals reflect the laws of organ movement. The fusion of the two can cover the full dimensional information of "static anatomy-dynamic movement";

[0019] 2. Ultrasound confidence and physiological signal confidence convert data quality into computable numerical indicators, directly determining the fusion weight distribution and ensuring that "high-confidence data dominates the results";

[0020] 3. Adjust the confidence level based on the rate of change of the deviation value and the deviation sensitivity coefficient to adjust the weight of the main mode, reducing the impact of low-quality data on the fusion results while strengthening the dominant role of valid data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0022] Figure 1 is a flow chart of the method of the present invention;

[0023] Figure 2 It is a data flow diagram of the present invention. DETAILED DESCRIPTION

[0024] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention and should not be regarded as the entire invention or as a limitation or restriction of the technical solution of the present invention.

[0025] Traditional methods for dynamic positioning of medical devices use a fixed weighting mechanism for multimodal data fusion. This mechanism is unable to adjust data confidence in real time based on the degree of intraoperative ultrasound image artifact interference and fluctuations in the signal-to-noise ratio of physiological signals. This results in low-quality modal data having a dominant influence on displacement prediction results. For example, when ultrasound images are distorted by vascular or calcification artifacts, or when physiological signals experience a sudden drop in signal-to-noise ratio due to motion artifacts, the fixed weighting will amplify the error contribution of low-confidence data, causing the final predicted displacement to deviate from the actual organ displacement trajectory.

[0026] For example, during radiofrequency ablation surgery for liver tumors, the ultrasound probe needs to be tilted to collect images due to rib obstruction, resulting in acoustic artifacts in the branches of the hepatic vein. This increases the error in extracting the contour of the lesion area in the ultrasound image. At the same time, the patient's respiratory rhythm disorder caused by pain leads to a decrease in the signal-to-noise ratio of the respiratory phase signal. If the system still fuses the ultrasonic displacement value and the physiological signal prediction value with a fixed weight, the ultrasound data under artifact interference will still be calculated with a high weight, and the prediction error of the low signal-to-noise ratio physiological signal will not be effectively suppressed, which will eventually cause the positioning of the ablation needle tip to deviate from the actual target position beyond the device's correction capability.

[0027] If these issues are not addressed, positioning errors will continue to accumulate during the dynamic displacement of organs, preventing the medical device from generating adjustment instructions that match the real-time anatomical state. In scenarios with periodic displacement caused by respiratory motion, the accumulation of errors will cause the device's actuators to respond with hysteresis, resulting in incomplete coverage of the ablation area or excessive damage to healthy tissue. During cardiac interventional procedures, such errors can directly lead to mechanical collisions between the catheter and the inner wall of the cardiac cavity, increasing the risk of surgical complications.

[0028] Faced with the aforementioned issues, this application first analyzed the root cause of the inability of traditional fixed-weight fusion mechanisms to cope with dynamic fluctuations in intraoperative data quality, discovering that the underlying issue lies in the lack of real-time confidence assessment and feedback adjustment capabilities for multimodal data. For scenarios involving ultrasound artifacts and sudden changes in the signal-to-noise ratio of physiological signals, two improvement approaches were explored: one, dynamic weight adjustment based on a single data quality metric (such as artifact area ratio), and the other, a closed-loop feedback mechanism linking multimodal confidence and device execution error. A comparative study revealed that while the former approach can adjust weights in real time based on data quality, it fails to consider the correlation between the actual device positioning performance and the predicted results, potentially leading to a mismatch between the weight adjustment direction and the error correction requirements. The latter approach, by reverse-mapping the device displacement deviation and its changing trend to the confidence adjustment strategy, can form an error-adaptive dynamic fusion mechanism. Based on this, this application chose to link confidence adjustment with the device's real-time displacement deviation and introduce the deviation change rate as a dynamic factor for weight correction, enabling the fusion strategy to autonomously optimize based on the error evolution trend.

[0029] In this regard, Figure 1 As shown, the present application proposes: a medical equipment automatic control and positioning system based on multimodal data, which runs a medical equipment automatic control and positioning method based on multimodal data during operation, and the method includes the following steps:

[0030] Acquire preoperative pre-positioning data and intraoperative ultrasound images, physiological signal data and equipment calibration data; the preoperative pre-positioning data includes the patient's three-dimensional image model; multimodal data refers to a heterogeneous information set including preoperative three-dimensional image models, intraoperative ultrasound images, physiological signals and equipment calibration data, which can be specifically implemented using a medical image archiving system and a real-time sensor acquisition module, and a complete intraoperative positioning information foundation is constructed by integrating anatomical structures, dynamic physiological parameters and equipment status data.

[0031] A unified timestamp is added to the intraoperative ultrasound images, physiological signal data, and equipment calibration data to generate multimodal data with time series alignment; unified timestamp refers to the time synchronization marking of data streams with different sampling frequencies during surgery, which can be specifically implemented using a network time protocol or a hardware clock synchronization module to ensure that the ultrasound image frames, physiological signal sampling points, and equipment displacement records have a unified time base.

[0032] Based on the artifact characteristics of the intraoperative ultrasound image and the signal-to-noise ratio of the physiological signal data, the ultrasound confidence and physiological signal confidence are calculated respectively. The ultrasound confidence is an indicator that quantifies the degree of artifact interference in the ultrasound image. Specifically, it can be achieved by using an image segmentation algorithm to identify the artifact areas of blood vessels and calcification foci and calculate the area ratio. The reliability of displacement extraction is improved by reducing the weight of the artifact-contaminated area. The physiological signal confidence is an evaluation parameter that reflects the quality of physiological signal data such as respiration and heartbeat. Specifically, it can be achieved by using a signal-to-noise ratio calculation module combined with an adaptive filtering algorithm. The input quality of the displacement prediction model is optimized by dynamically evaluating signal clarity.

[0033] Using the time-aligned physiological signal data, the displacement prediction model generates the predicted value of the organ's dynamic displacement, and the actual displacement value of the organ is extracted based on the intraoperative ultrasound image; the displacement prediction model refers to an algorithm that infers the dynamic displacement of the organ based on the timing characteristics of the physiological signal. Specifically, it can be implemented using a Kalman filter or a recurrent neural network model, and the predicted value is generated by establishing a nonlinear mapping relationship between the respiratory phase, heart cycle and organ movement.

[0034] According to the ultrasound confidence level and the physiological signal confidence level, the actual displacement value of the ultrasound image and the dynamic displacement prediction value are weightedly fused to obtain the final predicted displacement. Weighted fusion refers to dynamically allocating the weights of the ultrasound measured displacement and the physiological signal predicted displacement according to the real-time confidence level. Specifically, this can be achieved using a linear weighted algorithm, and positioning deviations caused by single modal failures are suppressed by prioritizing the use of high-confidence data sources.

[0035] The real-time displacement information of the device is extracted using the time-aligned device calibration data to obtain the deviation value between the final predicted displacement and the device's real-time displacement information. The deviation value refers to the difference between the final predicted displacement and the actual device displacement. Specifically, it can be implemented using the Euclidean distance calculation module. The confidence adjustment strategy is triggered through the real-time feedback control mechanism to maintain system stability.

[0036] Determine whether the deviation value exceeds the preset deviation threshold. If so, perform the following operations:

[0037] The ultrasound confidence and physiological signal confidence are adjusted based on the rate of change of the deviation value. The preset deviation threshold refers to the maximum allowable displacement error set according to the type of surgery. It can be specifically implemented by matching the clinical safety standard database, and the need for iterative optimization of the confidence parameters is determined by threshold comparison.

[0038] Based on the adjusted ultrasound confidence level and physiological signal confidence level, a reweighted fusion calculation is performed to predict displacement values, and adjustment instructions for the medical device are generated based on the predicted displacement values. Confidence adjustment dynamically corrects the weighting of ultrasound and physiological signal data sources based on the rate of change of deviations. This is achieved using a coupling algorithm of normalization coefficients and sensitivity coefficients. By introducing deviation trend prediction, the system's response speed to sudden interference is improved.

[0039] The core innovation of this application lies in the construction of a dynamic confidence adjustment mechanism. By real-time evaluation of data quality indicators such as ultrasound artifact area and physiological signal-to-noise ratio, combined with the changing trends of displacement prediction deviations, the fusion weights of multimodal data are dynamically optimized. This mechanism effectively solves the problem of low-quality data dominance that occurs in traditional fixed-weight fusion methods when the intraoperative environment suddenly changes, significantly improving the robustness and positioning accuracy of dynamic organ displacement prediction.

[0040] like Figure 2 As shown, this is the data flow of this application;

[0041] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0042] During radiofrequency ablation of liver tumors, a preoperative CT 3D image model of the patient is first obtained as pre-positioning data. During the procedure, ultrasound images, respiratory waveforms, and electrocardiogram signals are collected as physiological signal data, while the position of the radiofrequency ablation needle is recorded as device calibration data.

[0043] Millisecond-level timestamps are added to the collected data to achieve time alignment of multimodal data. Image segmentation algorithms are used to identify artifact areas in ultrasound images, calculate the artifact area ratio, and obtain ultrasound confidence. De-noising is performed on respiratory waveforms and electrocardiogram signals, and the signal-to-noise ratio is calculated to obtain physiological signal confidence.

[0044] The Kalman filter model uses respiratory phase and heart cycle signals as input to predict the dynamic displacement of liver tumors. Simultaneously, the actual displacement of the tumor is calculated by registering the ultrasound image with the feature points of the preoperative CT model.

[0045] Based on the ultrasound confidence level and the physiological signal confidence level, the actual displacement value and the predicted displacement value are weighted and fused to obtain the final predicted displacement. The final predicted displacement is compared with the real-time position of the radiofrequency ablation needle to calculate the deviation value.

[0046] If the deviation exceeds a preset threshold, such as 3 mm, the ultrasound confidence level and the physiological signal confidence level are adjusted based on the rate of change of the deviation. Specifically, a confidence ratio coefficient is calculated and the confidence level is modified based on the rate of change of the deviation. The adjusted confidence level is used to recalculate the predicted displacement value and generate adjustment instructions for the RF ablation needle.

[0047] Through the above scheme, this application realizes the precise positioning and adjustment of multimodal data automatic control of medical equipment. By dynamically adjusting the weights of different data sources, the problems of intraoperative ultrasound artifacts and fluctuations in physiological signal quality can be effectively addressed. The closed-loop feedback mechanism can adaptively adjust the fusion strategy according to the real-time positioning deviation, thereby improving the accuracy and stability of device positioning. This method can reduce error accumulation and avoid the delayed response of the device actuator in the scenario of periodic organ displacement caused by respiratory movement, thereby improving the safety of surgery and the treatment effect.

[0048] This application further proposes a method for dynamically adjusting confidence based on the rate of change of the deviation value, including:

[0049] Get the current time point Ultrasound confidence , physiological signal confidence , deviation value and the rate of change of the deviation value , where the rate of change ;

[0050] Calculate the confidence ratio coefficient, ;

[0051] Ultrasound confidence adjustment is: ,in The current time point Deviation sensitivity coefficient;

[0052] The confidence of physiological signals is adjusted to: .

[0053] Rate of change of deviation value By calculating the deviation value of two consecutive time points, the instantaneous change trend of the deviation can be reflected. Confidence of physiological signals As the denominator, ultrasound confidence As a molecule, characterized by ultrasound confidence Confidence of physiological signals The relative advantage of bias sensitivity coefficient To control the rate of change sensitivity.

[0054] For example, in the case of a sudden increase in artifacts caused by poor contact of the ultrasound probe, the ultrasound confidence is 0.5, physiological signal confidence When the deviation value is 0.5 When the change rate increases from 0.8mm to 1.5mm within 0.5 seconds, The deviation sensitivity coefficient is 0.7 mm / s. is 0.8.

[0055] The adjusted ultrasound confidence level is 0.5*1*(1-0.7*0.8)=0.22, and the physiological signal confidence level is adjusted to 0.78. and deviation sensitivity coefficient The multiplication factor of makes the confidence adjustment amplitude positively correlated with the error change rate, and automatically reduces the weight of low-quality data when the deviation increases rapidly to avoid error accumulation.

[0056] Through the above technical solution, the present application achieves dynamic adjustment of ultrasound confidence and physiological signal confidence based on the rate of change of the deviation value. When the rate of change of the deviation value is large, the ultrasound confidence will decrease accordingly, and the physiological signal confidence will increase accordingly, making the subsequent fusion calculation more dependent on the physiological signal prediction results. Conversely, when the rate of change of the deviation value is small, the ultrasound confidence will increase relatively. This dynamic adjustment mechanism improves the accuracy and robustness of multimodal data fusion, enabling the system to better adapt to changes in the intraoperative environment and improve the positioning accuracy of medical equipment.

[0057] This application further proposes that the calculation method of the deviation sensitivity coefficient includes:

[0058] Get the deviation sensitivity coefficient at the previous time point and preset deviation thresholds ;

[0059] Deviation sensitivity coefficient The calculation formula is:

[0060] .

[0061] Among them, the deviation sensitivity coefficient The calculation is done by introducing the deviation sensitivity coefficient of the previous time point , current deviation value Deviation from preset threshold The ratio and confidence difference Dynamic adjustment in three dimensions.

[0062] Deviation sensitivity coefficient at the previous time point As a base value, the current deviation value Deviation from preset threshold The proportional relationship reflects the degree of deviation and the confidence difference The reliability difference between the two confidence data sources is compensated. When the confidence of ultrasound and physiological signals is close, the confidence difference Approaching 1, the deviation sensitivity coefficient Mainly affected by the degree of deviation; when the difference between the two confidence levels is significant, the confidence difference Approaching 0, reducing the deviation sensitivity coefficient The adjustment range is set to avoid overshoot.

[0063] For example: During cardiac catheterization, respiratory motion causes the organ to deviate from the preset threshold is 15mm, the current ultrasound confidence 0.7, physiological signal confidence is 0.3, the absolute value of the confidence difference 0.4, the deviation sensitivity coefficient at the previous time point is 0.8, the current deviation value 18mm;

[0064] The current deviation sensitivity coefficient The value of is 0.8*(18 / 15)*(1-0.4)=0.8*1.2*0.6=0.576. This coefficient ensures the continuity of adjustment by inheriting historical values. It amplifies the adjustment range when the deviation value exceeds the threshold, while suppressing the adjustment intensity due to the large difference in the reliability of the two data sources.

[0065] In subsequent confidence adjustments, the lower bias sensitivity coefficient Make ultrasound confidence The adjustment range is reduced to avoid the fluctuation of fusion results caused by the sudden drop in the reliability of a single data source. For example, when the ultrasound image has a vascular artifact that causes the ultrasound confidence to drop When the confidence level of physiological signals drops to 0.5, By increasing the synchronization to 0.5, the system can automatically balance the contribution weights of the two data sources through the adjustment of the confidence difference item.

[0066] Through the above technical solution, the present application realizes the dynamic adjustment of the deviation sensitivity coefficient. This method takes into account the proportional relationship between the current deviation value and the preset threshold, as well as the difference in confidence between ultrasound and physiological signals, and can adaptively adjust the sensitivity according to the real-time data quality. This dynamic adjustment mechanism improves the responsiveness to deviation changes and helps to maintain stable positioning accuracy under different surgical stages and data quality conditions. At the same time, by introducing the confidence difference factor, this method can also balance the influence of different modal data and avoid the excessive impact of fluctuations in the quality of a single modal data on the overall positioning accuracy.

[0067] This application further proposes that the calculation method of ultrasound confidence includes:

[0068] Identify artifact areas in ultrasound images using image segmentation algorithms;

[0069] Calculate the artifact area ratio based on the pre-stored artifact feature library ;

[0070] Ultrasound confidence The calculation formula is: ;

[0071] The artifact feature library contains artifact morphological features of blood vessels and calcification foci in target organs in ultrasound.

[0072] The image segmentation algorithm utilizes a deep learning model based on the U-Net architecture. Its input is an ultrasound image, and its output is a binary mask of the artifact area. The artifact feature library stores the spatial distribution patterns of morphological features such as vascular branch intersections and speckled hyperechoic calcifications, serving as a comparison benchmark for artifact identification. The artifact area percentage is calculated by calculating the ratio of the total number of mask pixels to the total number of pixels in the target organ area, with this ratio constrained to between 0 and 1.

[0073] Specifically, in the ultrasound image processing stage, the real-time ultrasound image is first input into the trained segmentation model. The segmentation model extracts image features through the convolution layer and matches them with the morphological template in the artifact feature library to output the artifact area boundary. Then, the area ratio of the artifact area in the target organ is calculated. When the ratio reaches 0.3, the ultrasound confidence level is high. Automatically reduced to 0.7. This calculation method quantifies the impact of artifacts generated by specific anatomical structures, effectively avoiding the misjudgment problem caused by the complex morphology of artifacts in traditional methods. For example, in liver surgery scenarios, the segmentation model can accurately distinguish the acoustic shadows of hepatic vein branches from the actual lesion area, ensuring that the confidence calculation only reflects the displacement information of valid anatomical structures.

[0074] For example, if the artifact area accounts for 30%, the ultrasound confidence is 0.7.

[0075] Through the above technical solution, the present application can dynamically calculate ultrasound confidence based on artifact features in ultrasound images, thereby providing a reliable weight basis for subsequent multimodal data fusion. This can avoid over-reliance on ultrasound data when ultrasound image quality is poor, improving the stability and reliability of overall positioning accuracy. Furthermore, by introducing a pre-stored artifact feature library, the present solution can accurately identify ultrasound artifact features for different organs and lesion types, improving the applicability and accuracy of ultrasound confidence calculation.

[0076] This application further proposes that the calculation method of physiological signal confidence includes:

[0077] Calculate the signal-to-noise ratio after denoising the physiological signal ;

[0078] Physiological signal confidence The calculation formula is: ,in Signal-to-noise ratio The preset maximum value.

[0079] Among them, the denoising process is realized by wavelet transform algorithm or adaptive filter, and the signal-to-noise ratio The signal-to-noise ratio is obtained by calculating the ratio of signal power to noise power. The preset maximum value of is set based on the best signal-to-noise ratio of physiological signals in historical surgical data. By normalizing the signal-to-noise ratio Mapped to the 0-1 interval, when the signal-to-noise ratio reaches the preset maximum value, the physiological signal confidence is 1, indicating that the signal quality is optimal.

[0080] Specifically, when denoising physiological signals, a Butterworth low-pass filter is used to filter out high-frequency myoelectric interference and retain low-frequency components related to breathing and heartbeat. The calculation is done by dividing the signal into 5-second time windows, and calculating the signal power spectrum density and noise power spectrum density in each time window. The noise power spectrum density is taken as the energy average outside the 0.5-1.5Hz frequency band. The preset maximum value is set to 40dB, which is determined based on the peak signal-to-noise ratio statistics of ECG signals in 100 clinical cases. With the preset maximum value Perform ratio calculation and physiological signal confidence When quantized to 0.75, it indicates that the current signal quality has reached 75% of the maximum signal-to-noise ratio. At this point, the physiological signal data occupies a corresponding weight in the fusion calculation. This calculation method ensures that when the physiological signal quality fluctuates, the confidence level can reflect its reliability changes in real time, thereby avoiding the negative impact of low-quality signals on displacement prediction.

[0081] Through the above technical solution, this application can dynamically adjust the confidence level of physiological signals based on their real-time quality, preventing low-quality signals from negatively impacting the fusion results. This improves the accuracy and reliability of multimodal data fusion and further enhances the positioning accuracy of medical devices. Furthermore, this method is computationally simple and fast, meeting the needs of real-time processing.

[0082] This application further proposes that the displacement prediction model is a Kalman filter model. The input of the displacement prediction model is the respiratory phase signal and the cardiac cycle signal in the time-aligned physiological signal data, and the output is the dynamic displacement prediction value of the organ at the current time point; the displacement prediction model is trained through historical surgical data to establish the correlation between the respiratory phase and the organ displacement, and the initial parameters include the process noise covariance matrix and the measurement noise covariance matrix.

[0083] Among them, the respiratory phase signal and the heart cycle signal are used as the input of the Kalman filter model. The periodic characteristics are extracted through time series analysis and dynamically mapped with the organ displacement. During the training process of historical surgical data, the least squares method is used to fit the nonlinear relationship between the respiratory phase angle and the organ displacement to generate the state transfer equation. The process noise covariance matrix is ​​used to quantify the uncertainty of the model prediction stage, and the measurement noise covariance matrix is ​​used to quantify the noise interference during the physiological signal acquisition process. The initial parameters are determined by offline training optimization. For example, in the historical data set, the process noise covariance matrix is ​​set to 0.01-0.1mm 2 / s, the measurement noise covariance matrix is ​​set to 0.05-0.2mm 2 .

[0084] Specifically, the Kalman filter model realizes dynamic displacement prediction through prediction steps and update steps. In the prediction step, the displacement prediction value at the current moment is calculated based on the displacement estimate value at the previous moment and the state transition equation of the respiratory phase signal, and the process noise covariance matrix is ​​superimposed to reflect the randomness of organ movement; in the update step, the actual measured physiological signal is compared with the predicted value, and the Kalman gain is calculated in combination with the measurement noise covariance matrix to correct the displacement prediction value. For example, when the signal-to-noise ratio of the respiratory signal drops sharply due to the patient's cough, the measurement noise covariance matrix increases, and the Kalman gain automatically decreases, reducing the impact of noise data on the prediction results. Furthermore, the initial value of the process noise covariance matrix is ​​calculated by the standard deviation of the organ displacement in the historical data. For example, when the standard deviation of liver displacement is 2.5mm, the initial value of the process noise covariance matrix is ​​set to 6.25mm. 2 By dynamically adjusting the weights of predicted and measured data, the Kalman filter model can maintain prediction stability when organ motion patterns change, thereby providing high-precision displacement prediction values ​​for subsequent weighted fusion.

[0085] Through the above technical solution, this application can accurately predict the dynamic displacement of organs. Using the Kalman filter model, respiratory phase and heart cycle signals are associated with organ displacement, overcoming the limitation of traditional static positioning models that cannot adapt to dynamic changes. At the same time, through historical data training and initial parameter setting, the adaptability and robustness of the model are improved. This dynamic prediction method provides a reliable foundation for subsequent precise positioning and adjustment of equipment, helping to improve the accuracy and safety of surgery.

[0086] This application further proposes that after the ultrasound confidence and physiological signal confidence are adjusted, the following are also included:

[0087] Calculating the adjusted deviation between the final predicted displacement and the actual displacement value based on the adjusted ultrasound confidence and physiological signal confidence;

[0088] Determine the deviation value before adjustment The deviation value after adjustment Is the ratio lower than the preset adjustment threshold? If so, perform the following operations:

[0089] Process noise covariance matrix of the modified Kalman filter model ,Adjustment ,in, is the original process noise covariance matrix.

[0090] The step of adjusting the process noise covariance matrix includes: after each confidence adjustment, calculating the deviation value before adjustment The deviation value after adjustment , when the ratio When it falls below the preset adjustment threshold, the process noise covariance matrix is ​​triggered Correction of process noise covariance matrix The correction range is limited to an upper limit of 1.4 to avoid drastic fluctuations in model parameters due to sudden changes in the deviation value.

[0091] Specifically, if the deviation value before adjustment The deviation value after adjustment The ratio is lower than the preset adjustment threshold, indicating that the adjusted deviation value Too large, that is, the adjustment of confidence has limited effect on improving the prediction accuracy. Therefore, by increasing the value of the process noise covariance matrix, the model can track the actual displacement changes more sensitively, improve the prediction accuracy, and be used for confidence evaluation or displacement prediction at subsequent time points.

[0092] For example, if the deviation value before adjustment is The deviation value after adjustment is 2.5mm is 1.8mm, the ratio is 1.39, and the preset adjustment threshold is 1.5; the correction operation is triggered.

[0093] Adjusted process noise covariance matrix for , making the model more sensitive to actual displacement changes and improving prediction accuracy. This mechanism enables the Kalman filter model to dynamically adapt to data fusion results after changes in confidence weights, improving the long-term stability of displacement prediction.

[0094] Through this adjustment, the prediction accuracy of the Kalman filter model is further optimized.

[0095] Through the above technical solution, the present application can dynamically adjust the Kalman filter model parameters according to the changes in the deviation value, improving the adaptability and accuracy of the displacement prediction model. As a result, the positioning accuracy of medical equipment is further improved, reducing the surgical risks caused by dynamic organ displacement.

[0096] The present application further proposes that the actual displacement value of the organ extracted from the intraoperative ultrasound image includes:

[0097] The actual displacement value is obtained by calculating the difference between the center point of the lesion in the ultrasound image and the center point of the lesion in the preoperative model after feature point registration. The registration algorithm is an affine transformation based on the feature points. After registration, the coordinate difference between the center point of the lesion in the ultrasound image and the center point of the lesion in the preoperative model is calculated.

[0098] The feature point registration process specifically includes the following implementation methods: selecting vascular bifurcation points and calcification edge points in the preoperative 3D imaging model as a set of baseline feature points; extracting the corresponding vascular bifurcation points and calcification edge points in the intraoperative ultrasound image using an edge detection algorithm as a set of real-time feature points; calculating the affine transformation matrix between the two sets of feature points using the least squares method, which contains translation, rotation, and scaling parameters; substituting the coordinates of the lesion center point in the preoperative model into the affine transformation matrix to obtain the theoretical coordinates after registration; and calculating the Euclidean distance between the actual lesion center point coordinates in the ultrasound image and the theoretical coordinates as the actual displacement value. For example, when the ultrasound image has a 15% artifact area, by selecting at least 6 pairs of non-collinear feature points for affine transformation, the registration error can be controlled within 0.8mm.

[0099] Specifically, during the surgical procedure, after acquiring ultrasound images in real time, the system first identifies the lesion region using an image segmentation algorithm and extracts its geometric center as the measured lesion center coordinates. Simultaneously, the original lesion center coordinates stored in the preoperative 3D imaging model are retrieved. A spatial mapping relationship between the two coordinate systems is established through feature point registration. The affine transformation matrix is ​​solved using the RANSAC algorithm to eliminate outliers and ensure registration stability even with some feature point mismatches. After the coordinate transformation, the system calculates the displacement components between the measured and theoretical points along three axes and uses vector synthesis to obtain the actual displacement values ​​in 3D space. This method's advantage lies in compensating for coordinate shifts caused by intraoperative organ deformation through geometric transformation. For example, when the liver undergoes volumetric changes due to respiratory motion, the scaling parameters in the affine transformation automatically adjust the model scale, ensuring that the registered lesion center coordinates are more consistent with the actual anatomical structure. Experimental data demonstrate that compared to traditional rigid registration methods, this approach reduces displacement calculation errors during respiratory phase changes by 42%.

[0100] Through the above technical solution, this application effectively solves the problem of positioning misalignment caused by dynamic displacement of organs during surgery. Through spatial mapping of feature point registration and affine transformation, dynamic alignment of ultrasound images with preoperative three-dimensional models is achieved, overcoming the defect of traditional static registration that cannot adapt to respiratory motion deformation. This technical means improves the calculation accuracy of actual displacement values ​​to the submillimeter level, ensuring that medical equipment can track the target position in real time, thereby reducing the risk of damage to surrounding healthy tissue caused by positioning errors.

[0101] This application further proposes weighted fusion of the actual displacement value of the ultrasound image and the dynamic displacement prediction value, including:

[0102] Get the current time point Ultrasound confidence , physiological signal confidence , actual displacement value , dynamic displacement prediction value ;

[0103] The calculation formula for the final predicted displacement is:

[0104] .

[0105] The calculation formula for the final predicted displacement adopts a weighted fusion method, where: and As the actual displacement value and dynamic displacement prediction values The design of this weighted fusion is based on the consideration that in the actual displacement measurement and prediction related to ultrasound images, there may be certain limitations in relying solely on actual measured values ​​or solely on predicted values. For example, in some cases, the actual measured values ​​of ultrasound images may be It may be inaccurate due to noise interference, image quality problems, etc., and if the dynamic displacement prediction value Based on a more reliable model and previous data, it can supplement and correct the final displacement prediction to a certain extent; on the contrary, when the dynamic displacement prediction value When there is a large deviation due to some sudden factors (such as sudden involuntary movement of the human body, etc.), the relatively accurate actual displacement value Then, the larger weight ( Relatively large) to ensure the final predicted displacement reliability.

[0106] As a preferred embodiment, the solution of this application is specifically implemented as follows:

[0107] Ultrasound confidence The confidence level of physiological signal is 0.513. The actual displacement value is 0.487. The predicted dynamic displacement value is 12.553 mm. is 13.451 mm, then the final predicted displacement is calculated as:

[0108] 0.513*12.553+0.487*13.451≈12.99.

[0109] This application further proposes a The ratio of the number of successful times dominated by ultrasound to the number of successful times dominated by physiological signals in the adjustment was used to calculate the reliability coefficient of ultrasound modality. , physiological modal reliability coefficient ;

[0110] The modified weighted fusion formula is:

[0111] .

[0112] The ultrasound-led success count is defined as the number of adjustments in which the deviation decreases after the adjustment and the ultrasound confidence weight is greater than the physiological signal confidence weight. The same judgment logic applies to the physiological signal-led success count. The ultrasound modal reliability coefficient is calculated as the ratio of the ultrasound-led success count to the total number of adjustments made in the previous N times. The physiological modal reliability coefficient is calculated similarly. The revised weighted fusion formula combines the long-term modal reliability with the current data quality. For example, when the ultrasound modal reliability coefficient is 1.2 and the physiological modal reliability coefficient is 0.9, the effective weight of the ultrasound confidence increases by 20%.

[0113] Specifically, after each adjustment instruction is generated, the system records whether the adjustment successfully reduces the deviation value and determines the dominant mode. If 3 of the first 5 adjustments are ultrasound-dominated and successful, the ultrasound modality reliability coefficient is 3 / 5=0.72, which is corrected to 1.15 after normalization. During weighted fusion, even if the current ultrasound confidence is temporarily reduced to 0.6 due to artifacts, its effective weight is still 1.15*0.6=0.69, which is higher than 0.6 when the reliability coefficient is not introduced. Therefore, in the stage where the organ displacement law is stable, the system can compensate for instantaneous data quality fluctuations through historical successful data, avoiding the deviation of weight distribution from the actual modality reliability due to single confidence abnormalities. For example, in cardiac catheter intervention, when the signal-to-noise ratio of the respiratory signal drops sharply due to poor electrode contact, the confidence of the physiological signal may drop from 0.8 to 0.5. However, if historical data shows that the physiological signal was successfully guided 4 out of the last 5 times, its reliability coefficient is corrected to 1.25 after normalization, then the effective weight is 1.25*0.5=0.625, which is still higher than the instantaneous confidence, maintaining the stability of the fusion result.

[0114] Through the above technical solution, the present application can dynamically modify the fusion weight distribution based on the historical adjustment success rate of different modalities, reducing the negative impact of low-reliability modalities on the final displacement prediction in scenarios where ultrasound image artifact interference persists. When a modality shows a higher success rate in multiple adjustments, its reliability coefficient will increase the weight ratio of the modality confidence, thereby forming a more optimal modal contribution distribution mechanism in the dynamic displacement prediction of organs, avoiding the problem of systematic error accumulation caused by fixed weight distribution.

[0115] The technical scope of the present invention is not limited to the contents of the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A medical equipment automated control and positioning system based on multimodal data, characterized by: A method for automatically controlling and positioning medical equipment based on multimodal data is run during operation, the method comprising the following steps: Acquiring preoperative pre-positioning data and intraoperative ultrasound images, physiological signal data, and equipment calibration data; the preoperative pre-positioning data includes a three-dimensional imaging model of the patient; adding a unified timestamp to the intraoperative ultrasound image, physiological signal data, and device calibration data to generate multimodal data with time series alignment; Calculating ultrasound confidence and physiological signal confidence based on artifact characteristics of the intraoperative ultrasound image and a signal-to-noise ratio of physiological signal data; Using the time-aligned physiological signal data, the displacement prediction model generates the predicted value of the organ's dynamic displacement, and the actual displacement value of the organ is extracted based on the intraoperative ultrasound image; According to the ultrasound confidence level and the physiological signal confidence level, weighted fusion is performed on the actual displacement value of the ultrasound image and the dynamic displacement prediction value to obtain a final predicted displacement; The real-time displacement information of the device is extracted using the time-aligned device calibration data to obtain the deviation value between the final predicted displacement and the real-time displacement information of the device; Determine whether the deviation value exceeds the preset deviation threshold. If so, perform the following operations: Adjusting ultrasound confidence and physiological signal confidence based on the rate of change of the deviation value; According to the adjusted ultrasound confidence level and physiological signal confidence level, a predicted displacement value is re-weighted and fused, and an adjustment instruction for the medical device is generated according to the predicted displacement value; Adjusting the ultrasound confidence and physiological signal confidence based on the rate of change of the deviation value includes: Get the current time point Ultrasound confidence , physiological signal confidence , deviation value and the rate of change of the deviation value , where the rate of change ; Calculate the confidence ratio coefficient , ; Ultrasound confidence adjustment is: ,in The current time point Deviation sensitivity coefficient; The confidence of physiological signals is adjusted to: ; The deviation sensitivity coefficient is calculated as follows: Determine whether the current time point is the initial time point; If the judgment result is yes, the staff will configure it based on historical experience; If the judgment result is no, calculate according to the following formula: ； in, is the deviation sensitivity coefficient at the previous time point, is the preset deviation threshold.

2. The medical equipment automated control and positioning system based on multimodal data according to claim 1, characterized in that: The ultrasound confidence level is calculated as follows: Identify artifact areas in ultrasound images using image segmentation algorithms; Calculate the artifact area ratio based on the pre-stored artifact feature library ; Ultrasound confidence The calculation formula is: ; The artifact feature library contains artifact morphological features of blood vessels and calcification foci in target organs in ultrasound.

3. The medical equipment automated control and positioning system based on multimodal data according to claim 1, characterized in that: The physiological signal confidence is calculated as follows: Calculate the signal-to-noise ratio after denoising the physiological signal ; Physiological signal confidence The calculation formula is: ,in Signal-to-noise ratio The preset maximum value.

4. The medical equipment automated control and positioning system based on multimodal data according to claim 1, characterized in that: The displacement prediction model is a Kalman filter model. The input of the displacement prediction model is the respiratory phase signal and the cardiac cycle signal in the time-aligned physiological signal data, and the output is the dynamic displacement prediction value of the organ at the current time point. The displacement prediction model is trained through historical surgical data to establish the correlation between the respiratory phase and the organ displacement. The initial parameters include the process noise covariance matrix and the measurement noise covariance matrix.

5. The medical equipment automated control and positioning system based on multimodal data according to claim 4, characterized in that: After the ultrasound confidence level and the physiological signal confidence level are adjusted, the following steps are also included: Calculating the adjusted deviation between the final predicted displacement and the actual displacement value based on the adjusted ultrasound confidence and physiological signal confidence; Determine the deviation value before adjustment The deviation value after adjustment Is the ratio lower than the preset adjustment threshold? If so, perform the following operations: Process noise covariance matrix of the modified Kalman filter model ,Adjustment ,in, is the original process noise covariance matrix.

6. The medical equipment automated control and positioning system based on multimodal data according to claim 1, characterized in that: The step of extracting the actual displacement value of the organ according to the intraoperative ultrasound image comprises: The actual displacement value is obtained by calculating the difference between the center point of the lesion in the ultrasound image and the center point of the lesion in the preoperative model after feature point registration. The registration algorithm is an affine transformation based on the feature points. After registration, the coordinate difference between the center point of the lesion in the ultrasound image and the center point of the lesion in the preoperative model is calculated.

7. The medical equipment automated control and positioning system based on multimodal data according to claim 1, characterized in that: The weighted fusion of the actual displacement value of the ultrasound image and the dynamic displacement prediction value includes: Get the current time point Ultrasound confidence , physiological signal confidence , actual displacement value , dynamic displacement prediction value ; The calculation formula for the final predicted displacement is: 。 8. The medical equipment automated control and positioning system based on multimodal data according to claim 7, characterized in that: The weighted fusion of the final predicted displacement also includes: Based on the previous The ratio of the number of successful times dominated by ultrasound to the number of successful times dominated by physiological signals in the adjustment was used to calculate the reliability coefficient of ultrasound modality. , physiological modal reliability coefficient ; The modified weighted fusion formula is: 。

Images