Intraoperative breathing follow-up method based on magnetic force-ultrasonic signal and related device

Abstract

The invention discloses an intraoperative breathing follow-up method based on a magnetic force-ultrasonic signal and a related device. The method comprises the following steps: synchronously acquiring magnetic induction and ultrasonic echo signals during operation; the magnetic induction signals are filtered and calculated to obtain body surface displacement, and the ultrasonic signal spots are tracked to obtain organ displacement; performing Kalman weighted fusion on the two types of displacements to form fused displacement data; on the basis of the fusion data, using an LSTM network to construct respiratory phase-organ displacement mapping; then, future organ displacement is predicted by combining mapping with a linear harmonic breathing physical equation, and a displacement compensation instruction is sent in real time. According to the invention, rapid high-precision real-time compensation is realized, and the surgical precision and safety are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of clinical medicine, artificial intelligence and robots, in particular to an intraoperative respiratory following method based on magnetic force-ultrasound signals and related devices. BACKGROUND

[0002] Current precise surgery of thoracic and abdominal tumors generally relies on intraoperative CT or X-ray intermittent imaging. Respiratory motion causes target displacement, and doctors can only rely on experience to pause breathing or repeat scanning to adjust the puncture path, which not only prolongs the anesthesia time but also increases the radiation dose, and still cannot avoid instrument deviation, positive cut margin or normal tissue damage. The existing commercial respiratory gating system only indirectly estimates the body surface infrared or pressure belt signal, which cannot synchronously reflect the real displacement of internal organs. Electromagnetic tracking can locate the instrument, but lacks real-time imaging of soft tissue. Ultrasound can dynamically observe organs, but is easily disturbed by gas and bone, and the single modality information island effect is obvious. The non-uniform time reference of multi-modal data, signal drift and noise accumulation also make the fusion accuracy decrease with the operation time, and there is an urgent need for a low-cost solution that can continuously, synchronously and accurately capture the "body surface-organ" displacement link during surgery to solve the problems of traditional gating lag, frequent image registration, high radiation and trauma. SUMMARY

[0003] In order to solve the above technical problems, the present application relates to an intraoperative respiratory following method based on magnetic force-ultrasound signals and related devices, including but not limited to an intraoperative respiratory following device based on magnetic force-ultrasound signals, an electronic device, a computer readable storage medium and a computer program product.

[0004] In a first aspect, an intraoperative respiratory following method based on magnetic force-ultrasound signals is provided, comprising the following steps: a. synchronously acquiring magnetic induction signals and ultrasound echo signals during operation; b. processing the magnetic induction signals to obtain body surface displacement information, and processing the ultrasound echo signals to obtain organ tissue displacement information; c. using Kalman filtering and weighted fusion algorithm to fuse the body surface displacement information and the organ tissue displacement information to obtain fused displacement data; d. using a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fused displacement data; e. predicting the organ displacement amount at a future time according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement amount.

[0005] According to any embodiment of the present application, the synchronous acquisition is based on a hardware trigger signal or a unified timestamp.

[0006] In combination with any of the embodiments of the present application, the processing of the magnetic induction signal includes filtering and solving the magnetic induction signal; and the processing of the ultrasonic echo signal includes speckle tracking or cross-correlation analysis of the ultrasonic echo signal.

[0007] In combination with any of the embodiments of the present application, the Kalman filter takes the body surface displacement information as an observation vector and the organ tissue displacement information as a state vector, and estimates the optimal displacement through a prediction-update cycle; and the weighted fusion algorithm assigns weights to the body surface displacement information and the organ tissue displacement information according to the signal-to-noise ratio.

[0008] In combination with any of the embodiments of the present application, the bidirectional LSTM network includes a forward layer and a backward layer connected in parallel, the forward layer reads the fused displacement data in a time-increasing order, the backward layer reads the fused displacement data in a time-decreasing order, and the hidden states are spliced through a fully connected layer to output an organ displacement curve corresponding to a respiratory phase; and the respiratory phase is extracted from the fused displacement data through Hilbert transform to obtain a sequence of instantaneous phase angles, which are normalized and used as network inputs.

[0009] In combination with any of the embodiments of the present application, the prediction of the organ displacement at the future time is performed in a manner combining LSTM and a respiratory physical equation; and the respiratory physical equation is based on a linear harmonic model, and the formula is: wherein is the displacement of the organ, t is a time variable, f is a respiratory frequency, is a rate of change of thoracic pressure, a is a displacement amplitude, b represents a sensitivity of the displacement of the organ to the rate of change of pressure, is a phase shift.

[0010] In a second aspect, an intraoperative respiratory follow-up device based on magnetic-ultrasonic signals is provided, which includes: a perception unit including: a. a body surface ultrasonic probe for intraoperative acquisition of ultrasonic echo signals; b. a wearable magnetic sensor array for intraoperative acquisition of magnetic induction signals; c. a time synchronizer for synchronizing the operation of the body surface ultrasonic probe and the wearable magnetic sensor array; a preprocessing unit for processing the magnetic induction signals to obtain body surface displacement information, and processing the ultrasonic echo signals to obtain organ tissue displacement information; a fusion unit for information fusion of the body surface displacement information and the organ tissue displacement information using Kalman filtering and a weighted fusion algorithm to obtain fused displacement data; ​The mapping unit is configured to construct a respiratory phase-organ displacement mapping relationship according to the fused displacement data by using a bidirectional LSTM network. The prediction unit is configured to predict an organ displacement amount at a future time according to the mapping relationship, and send a displacement compensation instruction based on the organ displacement amount.

[0011] In a third aspect, an electronic device is provided, comprising a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is configured to store computer program code, the program code comprising computer instructions. When the processor executes the instructions, the electronic device will perform the method described in the second aspect above and any of its implementation manners.

[0012] In a fourth aspect, another electronic device is provided, comprising a processor, a wireless communication module, a touch screen, a speaker, and a storage unit, wherein the storage unit is configured to store computer program code, the program code comprising computer instructions. When the processor executes the instructions, the electronic device will perform the method described in the second aspect above and any of its implementation manners.

[0013] In a fifth aspect, a computer readable storage medium is provided, wherein the storage medium stores a computer program, the program comprising program instructions. When the instructions are executed by a processor, the processor will perform the method described in the second aspect above and any of its implementation manners.

[0014] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are run on a computer, the computer will perform the method described in the second aspect above and any of its implementation manners.

[0015] In the present application, compared with the prior art, an intraoperative respiratory following method based on magnetic force-ultrasound signals and related devices are provided. By synchronously collecting magnetic induction and ultrasound echo signals during surgery, body surface displacement information and organ tissue displacement information are extracted, respectively, and then Kalman filtering and weighted fusion are performed to generate unified fused displacement data. Subsequently, an LSTM network is used to establish a respiratory phase-organ displacement mapping, and a linear harmonic respiratory physical equation is combined to predict the organ displacement at a future time. Finally, displacement compensation instructions are output in real time. This method breaks through the limitations of traditional gating which only relies on body surface signals or a single image modality. For the first time, magnetic-ultrasound cross-modality information is synchronously coupled under hardware triggering or unified timestamps. By using a deep network and a physical equation collaborative prediction mechanism, continuous perception, mapping, and compensation of the whole chain of respiratory motion are achieved, significantly reducing the dependence on repeated intraoperative images, reducing radiation and anesthesia time, and improving surgical safety and comfort. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the accompanying drawings needed to be used in the embodiments of the present application or the background art will be described below.

[0017] The accompanying drawings incorporated into the specification and constituting a part of the specification illustrate the embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0018] Figure 1 A flowchart of an intraoperative respiratory following method based on magnetic force-ultrasound signal is proposed for the embodiments of the present application.

[0019] Figure 2 A front view of a wearable magnetic sensor array is proposed for the embodiments of the present application.

[0020] Figure 3 A curved side view of a wearable magnetic sensor array is proposed for the embodiments of the present application.

[0021] Figure 4 A schematic diagram of an intraoperative respiratory following device based on magnetic force-ultrasound signal is proposed for the embodiments of the present application.

[0022] Figure 5 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0023] In order to let the professionals in the technical field more fully understand the technical solutions of the present application, the technical solutions of the present application will be explained in detail and clearly by the accompanying drawings. It needs to be specially pointed out that the described embodiments are only part of the examples of the present application, and do not represent the whole. Based on these embodiments, the skilled in the art can directly deduce all other possible implementation schemes without creative thinking, and these are also included in the protection scope of the present application.

[0024] In the specification, claims and related drawings of the present application, the terms "first", "second", etc. are only used to distinguish different elements, and do not imply any specific order. At the same time, the use of "include" and "have" and their variants means non-exclusive inclusion. This means that if a process, method, system, product or device includes a series of steps or components, it means that the process, method, system, product or device is not limited to the listed steps or components, and can also include other steps or components not listed, or other steps or units inherent to it.

[0025] The term "embodiment" mentioned in this document refers to any example that combines specific features, structures or properties, which can belong to at least one embodiment of the present application. The term "embodiment" mentioned in this document does not necessarily refer to the same specific case, nor does it mean that they are mutually independent or alternative solutions. Those skilled in the art should understand that the embodiments described in this document can be used together with other embodiments. It should be clear that in this application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" means two or more instances.

[0026] It should be understood that the method embodiments of the present application can also be implemented by a processor executing computer program code. The embodiments of the present application are described below in conjunction with the accompanying drawings of the embodiments of the present application.

[0027] Please refer to Figure 1 , Figure 1 A flowchart of an intraoperative respiratory following method based on magnetic force-ultrasound signal is proposed for the embodiments of the present application.

[0028] 101, information acquisition: synchronously acquiring magnetic induction signals and ultrasound echo signals during operation.

[0029] In this embodiment, the ultrasound echo signals are acquired using a body surface ultrasound probe.

[0030] In this embodiment, the magnetic induction signals are acquired using a wearable magnetic sensor array, as shown in Figure 2 、 Figure 3 .

[0031] In this embodiment, the synchronous acquisition is achieved by a hardware trigger signal or a unified timestamp.

[0032] In this embodiment, the hardware trigger signal is generated by an FPGA and simultaneously output to the magnetic induction acquisition module and the ultrasound acquisition module.

[0033] In this embodiment, the unified timestamp adopts the PTP protocol, and both the magnetic induction signals and the ultrasound echo signals are labeled with absolute timestamps to prevent phase errors caused by clock drift.

[0034] In another possible implementation, the synchronous acquisition is completed by the IEEE1588 precision clock protocol and the hardware trigger signal, and both the magnetic induction and ultrasound acquisition ends are configured with PTP slave clocks.

[0035] In another possible implementation, the synchronous acquisition adopts a GNSS time service module to output a PPS second pulse, which is frequency-divided to simultaneously trigger the magnetic induction acquisition unit and the ultrasound acquisition unit, ensuring the consistency of the absolute time reference across devices.

[0036] In another possible implementation, the wearable magnetic sensor array is added to a respiratory detection belt to obtain signals of mechanical modalities.

[0037] 102. Preprocessing: processing the magnetic induction signals to obtain body surface displacement information, and processing the ultrasound echo signals to obtain organ tissue displacement information.

[0038] In this embodiment, the processing of the magnetic induction signals includes Butterworth low-pass filtering and magnetic dipole calculation, the filtering is used to suppress high-frequency electromagnetic noise, and the calculation obtains three-dimensional coordinates of the magnetic source by least squares fitting.

[0039] In this embodiment, the processing of the ultrasound echo signals selects a speckle tracking algorithm, which uses block matching and normalized cross-correlation function to track the displacement of tissue speckles over time.

[0040] In another possible implementation, a YOLO target detection algorithm is selected to process the ultrasound echo signals, and the version of the YOLO is not less than YOLOv5.

[0041] In another possible implementation, a cross-correlation analysis algorithm is selected to process the ultrasound echo signals, which calculates the cross-power spectrum of adjacent frame complex signals in the frequency domain and inversely transforms to obtain the displacement field.

[0042] In another possible implementation, Kalman filtering is used to process the magnetic induction signals to estimate signals and noise simultaneously in a state space model, thereby improving dynamic accuracy.

[0043] In another possible implementation, an optical flow method is used to process the ultrasound echo signals, which tracks the displacement of tissue feature points by Lucas-Kanade sparse optical flow, thereby reducing the calculation amount of speckle tracking.

[0044] In another possible implementation, a deep convolutional neural network regression is used to process the ultrasound echo signals, which directly outputs displacement vectors, the network takes a sequence of B-mode images as input, and is trained end-to-end without explicit speckle hypothesis.

[0045] 103、information fusion: using a weighted fusion algorithm, extracting the variance inverse, normalizing the signal-to-noise ratio and the normalized cross-correlation coefficient of the body surface displacement information and the organ tissue displacement information, multiplying the normalized signal-to-noise ratio and the normalized cross-correlation coefficient after sigmoid mapping as the credibility, multiplying the credibility and the variance inverse to obtain the weight, and normalizing the weight to obtain the weighted fusion data; Kalman filtering fusion is performed on the body surface displacement information and the organ tissue displacement information to obtain Kalman fusion data; the weighted fusion data and the Kalman fusion data are combined to obtain fusion displacement data.

[0046] In this embodiment, the Kalman filter takes the body surface displacement information as the observation vector and the organ tissue displacement information as the state vector to estimate the optimal displacement through a prediction-update cycle.

[0047] In this embodiment, the combination of the weighted fusion data and the Kalman fusion data adopts Bayesian model averaging, and the weighted fusion data and the Kalman fusion data are used as candidate models, and the weighted sum is obtained according to the posterior probability.

[0048] In another possible implementation, the information fusion adopts a particle filter framework, taking the body surface displacement as the observation and the organ displacement as the hidden state, and realizing robust estimation under non-Gaussian noise through resampling and weight updating.

[0049] In another possible implementation, the information fusion adopts adaptive weighted averaging, and the weight is dynamically adjusted through mutual information of the body surface displacement information and the organ tissue displacement information estimated in real time, so as to ensure that the high-confidence channel occupies a larger proportion.

[0050] In another possible implementation, the combination of the weighted fusion data and the Kalman fusion data adopts other combination methods, such as weighted averaging, covariance cross filtering, stacked generalization, etc.

[0051] 104、mapping establishment: using a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fusion displacement data.

[0052] In this embodiment, the bidirectional LSTM network is formed by connecting a forward layer and a backward layer in parallel, the forward layer reads the fusion displacement data in time increasing order, the backward layer reads the fusion displacement data in time decreasing order, and the two layers of hidden states are spliced through a full connection layer and then output the organ displacement curve corresponding to the respiratory phase; the respiratory phase is extracted from the fusion displacement data by Hilbert transform to obtain an instantaneous phase angle sequence, and the phase angle is normalized as network input, and the network training loss function adopts weighted mean square error, and different weights are given to the exhalation phase and the inhalation phase respectively to highlight the features of the rapid change section.

[0053] In another possible implementation, the loss function is trained using other networks.

[0054] In another possible implementation, the construction of the respiratory phase-organ displacement mapping relationship is implemented using a transformer-based network.

[0055] In another possible implementation, the construction of the respiratory phase-organ displacement mapping relationship adopts a temporal convolution network (TCN) whose dilated convolution structure can capture multi-scale respiratory cycle features while maintaining the advantage of parallel computing.

[0056] In another possible implementation, the construction of the respiratory phase-organ displacement mapping relationship adopts Gaussian process regression, taking respiratory phase as input and displacement as output, learning nonlinear mapping through kernel function and outputting uncertainty estimation.

[0057] 105、Displacement prediction: predicting the organ displacement amount at a future time according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement amount.

[0058] In this embodiment, the prediction adopts a combination of an LSTM network and a linear harmonic respiratory physical equation, and the LSTM network outputs a preliminary displacement sequence.

[0059] In this embodiment, the linear harmonic respiratory physical equation uses a sinusoidal harmonic superposition chest pressure rate of change model to physically constrain and correct the sequence, and the formula is: , wherein is the displacement amount of the organ, t is the time variable, f is the respiratory frequency, is the rate of change of chest pressure, a is the displacement amplitude, b represents the sensitivity of organ displacement to the rate of change of pressure, is the phase shift, and finally the organ displacement amount at a future time is obtained, and a displacement compensation instruction is triggered to a surgical robot or a treatment bed.

[0060] In another possible implementation, the prediction adopts an extended Kalman filter to fuse the LSTM network output and the respiratory physical equation, and uses a nonlinear state transition model to correct the prediction error in real time.

[0061] In another possible implementation, the prediction adopts a model predictive control framework, takes the LSTM network output as a system model, optimizes the displacement trajectory in a plurality of future time windows in a rolling manner, and issues the optimal control amount as a displacement compensation instruction.

[0062] In some embodiments, the apparatus provided by the embodiments of the present application has functions or includes modules that can be used to perform the methods described in the above method embodiment, and the specific implementation can refer to the description of the above method embodiments. For brevity, they will not be repeated here.

[0063] The above describes the method of the embodiments of the present application in detail. The apparatus of the embodiments of the present application is provided below.

[0064] Please refer to Figure 4 , Figure 4 A schematic diagram of an intraoperative respiratory follow-up device based on magnetic-ultrasound signals is provided for the embodiments of the present application. The respiratory follow-up device 1 includes a sensing unit 11, a preprocessing unit 12, a fusion unit 13, a mapping unit 14, and a prediction unit 15, specifically: The sensing unit 11 includes: a. a body surface ultrasound probe for intraoperative acquisition of ultrasound echo signals; b. a wearable magnetic sensor array for intraoperative acquisition of magnetic induction signals. For details, please refer to Figure 2 , Figure 3 ; C. a time synchronizer for synchronizing the operation of the body surface ultrasound probe and the wearable magnetic sensor array; The preprocessing unit 12 is used to process the magnetic induction signals to obtain body surface displacement information and process the ultrasound echo signals to obtain organ tissue displacement information; The fusion unit 13 is used to use Kalman filtering and weighted fusion algorithm to fuse the body surface displacement information and the organ tissue displacement information to obtain fusion displacement data; The mapping unit 14 is used to use a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fusion displacement data; The prediction unit 15 is used to predict the organ displacement amount at a future time according to the mapping relationship, and send a displacement compensation instruction based on the organ displacement amount.

[0065] In some embodiments, the apparatus provided by the embodiments of the present application has functions or includes modules that can be used to perform the methods described in the above method embodiment, and the specific implementation can refer to the description of the above method embodiments. For brevity, they will not be repeated here.

[0066] Please refer to Figure 5 , Figure 5A hardware architecture schematic of an electronic device is shown in the embodiments of the present application. The electronic device 2 is mainly composed of a processor 21 and a memory 22. In addition, the device can also include an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are connected to each other through connection components, which can be various interfaces, data lines or communication buses, etc. The embodiments of the present application do not make specific provisions for this.

[0067] The processor 21 can be one or more graphic processors (GPU). If the processor 21 is a GPU, the GPU can be single-core or multi-core. As an option, the processor 21 can also be composed of a processor group of multiple GPUs, which are connected to each other through one or more buses. In addition, the processor can also be other types of processors, which are not specifically limited in the embodiments of the present application.

[0068] The memory 22 is designed to save the instructions of the computer program and various program codes required for the implementation of the scheme of the present application. As an option, the memory can include but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or compact disc read-only memory (CD-ROM), which are used to store relevant instructions and data.

[0069] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices, or they can be an integral device.

[0070] It should be recognized that, in the embodiments of the present application, the memory 22 can not only save relevant instructions, but also save relevant data. The specific data content stored in the memory is not specifically limited in the embodiments of the present application.

[0071] It should be understood that, Figure 5 Only a simplified design of an electronic device is shown. In actual use, the electronic device can also include other necessary components, such as different numbers of input / output devices, processors, memories, etc. All electronic devices capable of implementing the embodiments of the present application belong to the protection scope of the present application.

[0072] Those skilled in the art should recognize that, according to the components and algorithm steps of various examples described in the embodiments disclosed herein, the functions can be realized by electronic hardware or in combination with computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application requirements and design constraints of the technical solutions. The skilled person can adopt different implementation methods according to the requirements of each specific application, but such implementation methods should not be regarded as beyond the protection scope of the present application.

[0073] It should be understood by those skilled in the art that, for the convenience of description and simplification of the description, the specific operation processes of the above system, device and component can refer to the corresponding steps in the foregoing method embodiments, which will not be repeated here. Meanwhile, professionals should also understand that each embodiment in the present application has its own emphasis, and for the convenience of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not described in detail in an embodiment, it can be referred to the relevant description of other embodiments.

[0074] In several embodiments provided in the present application, it should be recognized that the disclosed system, device and method can also be implemented by other ways. For example, the described device embodiments are only exemplary, and the division of the units is only logical functional division, and different division methods can exist in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be omitted, or some steps can not be performed. In addition, the connection between the units shown or discussed, whether direct or indirect, whether coupled or communicatively connected, can be realized through interfaces, devices or units in electrical, mechanical or other forms.

[0075] The units described as independent components can actually be physically separated or not; the parts presented as units can be physical entities or not, that is, they can be concentrated in one location or dispersed on multiple network nodes. According to the actual needs, part or all of these units can be selected to achieve the goal of the present embodiment.

[0076] Moreover, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each independent physical unit, or two or more units can be combined into one unit. In the foregoing embodiments, the related functions can be fully or partially implemented by software, hardware, firmware or any combination thereof. If software implementation is chosen, it can be implemented in whole or in part in the form of a computer program product. The computer program product contains one or more computer instructions. When the instructions are loaded and executed on a computer, they produce all or part of the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, DSL) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server, data center, etc. integrated with one or more available media. These available media can include magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art should understand that all or part of the processes of the above-mentioned embodiments can be completed by computer program instruction related hardware, and these programs can be stored in a computer-readable storage medium. When these programs are executed, they will contain the processes of the above-mentioned embodiments. The above-mentioned storage medium includes but is not limited to read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. A method for intraoperative respiratory tracking based on magnetic-ultrasonic signals, characterized in that: The steps include: a. Synchronously acquire magnetic induction signals and ultrasound echo signals during surgery; b. Processing the magnetic induction signal to obtain surface displacement information, processing the ultrasonic echo signal to obtain organ tissue displacement information; c. Using a weighted fusion algorithm, extract the inverse variance, normalized signal-to-noise ratio, and normalized cross-correlation coefficient from the body surface displacement information and the organ and tissue displacement information; multiply the normalized signal-to-noise ratio and the normalized cross-correlation coefficient after sigmoid mapping to obtain a credibility; perform a dot multiplication of the credibility with the inverse variance and normalize the result to obtain a weight; perform a weighted summation of the body surface displacement information and the organ and tissue displacement information according to the weight to obtain weighted fusion data; perform Kalman filtering on the body surface displacement information and the organ and tissue displacement information to obtain Kalman fusion data; and combine the weighted fusion data with the Kalman fusion data to obtain fused displacement data; d. Using a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship based on the fusion displacement data; e. Predicting the organ displacement at a future time according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement.

2. The method according to claim 1, characterized in that The synchronous acquisition is based on a hardware trigger signal or a unified timestamp.

3. The method according to claim 1, characterized in that The processing of the magnetic induction signal includes filtering and resolving the magnetic induction signal; and the processing of the ultrasonic echo signal includes performing speckle tracking or cross-correlation analysis on the ultrasonic echo signal.

4. The method according to claim 1, wherein The Kalman filter uses the body surface displacement information as the observation vector and the organ tissue displacement information as the state vector, and estimates the optimal displacement through a prediction-update cycle; the weighted fusion data and the Kalman fusion data are combined and averaged using a Bayesian model, and the weighted fusion data and the Kalman fusion data are used as candidate models, and weighted summation is performed according to the posterior probability.

5. The method according to claim 1, wherein The bidirectional LSTM network includes a forward layer and a backward layer connected in parallel. The forward layer reads the fused displacement data in ascending time order, and the backward layer reads the fused displacement data in descending time order. The hidden states are spliced ​​by a fully connected layer and then output as an organ displacement curve corresponding to the respiratory phase. The respiratory phase extracts an instantaneous phase angle sequence from the fused displacement data through Hilbert transform, and the phase angle is normalized and used as the network input.

6. The method according to claim 1, characterized in that The prediction of the organ displacement at the future moment is performed by combining a long short-term memory network model with a respiratory physics equation; the respiratory physics equation is based on a linear harmonic model, and the formula is: , in is the displacement of the organ, t is the time variable, f is the respiratory frequency, is the rate of change of thoracic pressure, a is the displacement amplitude, b represents the sensitivity of organ displacement to the rate of pressure change, is the phase offset.

7. A device for intraoperative respiratory tracking based on magnetic-ultrasonic signals, characterized in that: include: The sensing unit includes: a. a body surface ultrasound probe, used to obtain ultrasound echo signals during surgery; b. Wearable magnetic sensor array, used to obtain magnetic induction signals during surgery; C. a time synchronizer, for synchronizing the operation of the surface ultrasound probe and the wearable magnetic sensor array; Preprocessing unit: used for processing the magnetic induction signal to obtain body surface displacement information, and processing the ultrasonic echo signal to obtain organ tissue displacement information; Fusion unit: used to use a weighted fusion algorithm to extract the inverse variance, normalized signal-to-noise ratio and normalized cross-correlation coefficient of the body surface displacement information and the organ tissue displacement information, multiply the normalized signal-to-noise ratio and the normalized cross-correlation coefficient after sigmoid mapping as credibility, perform dot multiplication of the credibility and the inverse variance and normalize them to obtain a weight, perform weighted summation of the body surface displacement information and the organ tissue displacement information according to the weight to obtain weighted fusion data; perform Kalman filtering fusion on the body surface displacement information and the organ tissue displacement information to obtain Kalman fusion data; combine the weighted fusion data with the Kalman fusion data to obtain fused displacement data; Mapping unit: used for constructing a respiratory phase-organ displacement mapping relationship based on the fused displacement data using a bidirectional LSTM network; Prediction unit: used for predicting the organ displacement at a future moment according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement.

8. An electronic device, characterized in that: include: A processor and a storage unit, the storage unit is used to store computer program code, the code includes computer instructions, when the processor executes these instructions, the electronic device performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images