Intraoperative respiratory following method based on magnetic-ultrasonic signal and related device

Abstract

The application discloses a method and related device for intraoperative respiratory following based on magnetic force-ultrasonic signals. The method comprises the following steps: synchronously acquiring magnetic induction and ultrasonic echo signals during operation; filtering and solving the magnetic induction signals to obtain body surface displacement, and tracking the ultrasonic signal spots to obtain organ displacement; fusing the two types of displacement into fusion displacement data through Kalman weighting; constructing a respiratory phase-organ displacement mapping based on the fusion data using an LSTM network; and predicting future organ displacement by combining the mapping with a linear harmonic respiratory physical equation, and issuing displacement compensation instructions in real time. The application realizes fast and high-precision real-time compensation, and significantly improves the operation accuracy and safety.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of clinical medicine, artificial intelligence and robots, in particular to an intraoperative respiratory following method based on magnetic-ultrasound signals and related devices. BACKGROUND

[0002] Current precise surgery of thoracic and abdominal tumors generally relies on intraoperative CT or X-ray intermittent imaging. Respiratory motion causes target displacement, and doctors can only rely on experience to pause breathing or repeat scanning to adjust the puncture path, which not only prolongs the anesthesia time but also increases the radiation dose, and still cannot avoid instrument deviation, positive margin or normal tissue damage. The existing commercial respiratory gating system only indirectly estimates the surface infrared or pressure belt signal, which cannot synchronously reflect the real displacement of internal organs. Electromagnetic tracking can locate the instrument, but lacks real-time imaging of soft tissue. Ultrasound can dynamically observe organs, but is easily disturbed by gas and bone, and the single modality information island effect is obvious. The non-uniform time reference of multi-modal data, signal drift and noise accumulation also make the fusion accuracy decrease with the operation time, and there is an urgent need for a low-cost solution that can continuously, synchronously and accurately capture the "surface-organ" displacement link during surgery to solve the problems of traditional gating lag, frequent image registration, high radiation and trauma. SUMMARY

[0003] To solve the above technical problems, the present application relates to an intraoperative respiratory following method based on magnetic-ultrasound signals and related devices, including but not limited to an intraoperative respiratory following device based on magnetic-ultrasound signals, an electronic device, a computer readable storage medium and a computer program product.

[0004] In a first aspect, an intraoperative respiratory following method based on magnetic-ultrasound signals is provided, including the following steps:

[0005] a. synchronously acquiring magnetic induction signals and ultrasound echo signals during operation;

[0006] b. processing the magnetic induction signals to obtain surface displacement information, and processing the ultrasound echo signals to obtain organ tissue displacement information;

[0007] c. using Kalman filtering and weighted fusion algorithm to fuse the surface displacement information and the organ tissue displacement information to obtain fused displacement data;

[0008] d. using a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fused displacement data;

[0009] e. predicting the organ displacement amount at a future time according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement amount.

[0010] According to any one of the embodiments of the present application, the synchronous acquisition is based on a hardware trigger signal or a unified timestamp.

[0011] According to any one of the embodiments of the present application, the processing of the magnetic induction signal comprises filtering and solving the magnetic induction signal; and the processing of the ultrasonic echo signal comprises speckle tracking or cross-correlation analysis of the ultrasonic echo signal.

[0012] According to any one of the embodiments of the present application, the Kalman filter takes the body surface displacement information as an observation vector and the organ tissue displacement information as a state vector, and estimates the optimal displacement through a prediction-update cycle; and the weighted fusion algorithm assigns weights to the body surface displacement information and the organ tissue displacement information according to a signal-to-noise ratio.

[0013] According to any one of the embodiments of the present application, the bidirectional LSTM network comprises a forward layer and a backward layer connected in parallel, the forward layer reads the fused displacement data in a time-increasing order, the backward layer reads the fused displacement data in a time-decreasing order, and the hidden states are spliced through a full connection layer to output an organ displacement curve corresponding to a respiratory phase; and the respiratory phase is extracted from the fused displacement data through Hilbert transform to obtain a sequence of instantaneous phase angles, which are normalized and used as network inputs.

[0014] According to any one of the embodiments of the present application, the prediction of the organ displacement at the future time is performed in a manner combining an LSTM with a respiratory physical equation; and the respiratory physical equation is based on a linear harmonic model, and the formula is:

[0015] ,

[0016] wherein is the displacement of the organ, t is a time variable, f is a respiratory frequency, is a rate of change of thoracic pressure, a is a displacement amplitude, b represents a sensitivity of the organ displacement to the rate of change of pressure, is a phase shift.

[0017] In a second aspect, an intraoperative respiratory following device based on magnetic force-ultrasonic signals is provided, comprising:

[0018] a perception unit comprising: a. a body surface ultrasonic probe for intraoperative acquisition of ultrasonic echo signals;

[0019] b. a wearable magnetic sensor array for intraoperative acquisition of magnetic induction signals;

[0020] c. a time synchronizer for synchronizing the operation of the body surface ultrasonic probe and the wearable magnetic sensor array;

[0021] The preprocessing unit is configured to process the magnetic induction signals to obtain body surface displacement information, and process the ultrasonic echo signals to obtain organ tissue displacement information.

[0022] The fusion unit is configured to use a Kalman filter and a weighted fusion algorithm to fuse the body surface displacement information and the organ tissue displacement information to obtain fused displacement data.

[0023] The mapping unit is configured to use a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fused displacement data.

[0024] The prediction unit is configured to predict an organ displacement amount at a future time according to the mapping relationship, and send a displacement compensation instruction based on the organ displacement amount.

[0025] In a third aspect, an electronic device is provided, which includes a processor, a communication module, a sensor, a user interface, and a storage unit. The storage unit is configured to store computer program code, which includes computer instructions. When the processor executes the instructions, the electronic device will perform the method described in the second aspect above and any of its implementation manners.

[0026] In a fourth aspect, another electronic device is provided, which includes a processor, a wireless communication module, a touch screen, a loudspeaker, and a storage unit. The storage unit is configured to store computer program code, which includes computer instructions. When the processor executes the instructions, the electronic device will perform the method described in the second aspect above and any of its implementation manners.

[0027] In a fifth aspect, a computer readable storage medium is provided, which stores a computer program. The program includes program instructions. When the instructions are executed by a processor, the processor will perform the method described in the second aspect above and any of its implementation manners.

[0028] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions. When the computer program or instructions are run on a computer, the computer will perform the method described in the second aspect above and any of its implementation manners.

[0029] In the present application, compared with the prior art, an intraoperative respiratory following method based on magnetic force-ultrasound signal and a related device are provided. By synchronously collecting magnetic induction and ultrasound echo signals during operation, body surface displacement information and organ tissue displacement information are extracted, and then Kalman filtering and weighted fusion are performed to generate unified fusion displacement data. Subsequently, an LSTM network is used to establish a respiratory phase-organ displacement mapping, and a linear harmonic respiratory physical equation is combined to predict the organ displacement at a future time. Finally, displacement compensation instructions are output in real time. This method breaks through the limitations of traditional gating relying only on body surface signals or a single imaging modality. It is the first time that magnetic-sound cross-modal information is synchronously coupled under hardware triggering or unified timestamp. By using the prediction mechanism of deep network and physical equation cooperation, continuous perception, mapping and compensation of the whole chain of respiratory motion are realized, which significantly reduces the dependence on repeated intraoperative images, reduces radiation and anesthesia time, and improves surgical safety and comfort. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings needed to be used in the embodiments of the present application or the background art will be described below.

[0031] The drawings herein are incorporated into the description and form a part of the description, which show embodiments consistent with the present application, and are used together with the description to illustrate the technical solutions of the present application.

[0032] Figure 1 A flowchart of an intraoperative respiratory following method based on magnetic force-ultrasound signal is provided for the embodiments of the present application.

[0033] Figure 2 A front view of a wearable magnetic sensor array is provided for the embodiments of the present application.

[0034] Figure 3 A curved surface view of a wearable magnetic sensor array is provided for the embodiments of the present application.

[0035] Figure 4 A schematic diagram of an intraoperative respiratory following device based on magnetic force-ultrasound signal is provided for the embodiments of the present application.

[0036] Figure 5 A structural schematic diagram of an electronic device is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0037] In order to enable a person skilled in the art to have a more comprehensive understanding of the technical solutions of the present application, the technical solutions of the present application will be described in detail and clearly with reference to the accompanying drawings. It should be particularly pointed out that the described embodiments are only part of the examples of the present application, and do not represent the whole. Based on these embodiments, those skilled in the art can directly deduce all other possible embodiments without creative thinking, and these are also included in the protection scope of the present application.

[0038] In the specification, claims and related drawings of the present application, the terms "first", "second" and the like are used only to distinguish different elements, and do not imply any specific order. At the same time, the use of "include" and "have" and their variants means non-exclusive inclusion. This means that if a process, method, system, product or device includes a series of steps or components, it means that the process, method, system, product or device is not limited to the listed steps or components, and can also include other steps or components not listed, or other steps or units inherent to it.

[0039] In this paper, "embodiment" refers to any example that combines specific features, structures or properties, which may belong to at least one embodiment of the present application. The "embodiment" mentioned in this paper does not necessarily refer to the same specific case, nor does it mean that they are mutually independent or alternative solutions. Those skilled in the art should understand that the embodiments described herein can be used with other embodiments. It should be clear that in the present application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" means two or more instances.

[0040] It should be understood that the method embodiments of the present application can also be realized by a processor executing computer program code. The embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application.

[0041] Please refer to Figure 1 , Figure 1 A flowchart of an intraoperative respiratory following method based on magnetic-ultrasound signals is proposed for the embodiments of the present application.

[0042] 101, information acquisition: intraoperative synchronous acquisition of magnetic induction signals and ultrasonic echo signals.

[0043] In the present embodiment, the ultrasonic echo signals are acquired using a body surface ultrasonic probe.

[0044] In the present embodiment, the magnetic induction signals are acquired using a wearable magnetic sensor array, as shown in Figure 2 、 Figure 3 .

[0045] In the embodiment, the synchronization acquisition is realized by a hardware trigger signal or a unified timestamp.

[0046] In the embodiment, the hardware trigger signal is generated by an FPGA and simultaneously output to the magnetic induction acquisition module and the ultrasonic acquisition module.

[0047] In the embodiment, the unified timestamp adopts a PTP protocol, and both the magnetic induction signal and the ultrasonic echo signal are marked with an absolute timestamp to prevent phase error caused by clock drift.

[0048] In another possible implementation, the synchronization acquisition is realized by the IEEE1588 precise clock protocol and the hardware trigger signal, and both the magnetic induction and ultrasonic acquisition ends are configured with PTP slave clocks.

[0049] In another possible implementation, the synchronization acquisition adopts a GNSS time service module to output a PPS second pulse, which is frequency-divided to simultaneously trigger the magnetic induction acquisition unit and the ultrasonic acquisition unit, thereby ensuring consistency of the absolute time reference across devices.

[0050] In another possible implementation, the wearable magnetic sensor array is added to a breathing detection belt to acquire signals of mechanical modes.

[0051] 102, preprocessing: processing the magnetic induction signal to obtain body surface displacement information, and processing the ultrasonic echo signal to obtain organ tissue displacement information.

[0052] In the embodiment, the processing of the magnetic induction signal includes Butterworth low-pass filtering and magnetic dipole calculation, the filtering is used to suppress high-frequency electromagnetic noise, and the calculation obtains three-dimensional coordinates of the magnetic source through least square fitting.

[0053] In the embodiment, the processing of the ultrasonic echo signal selects a speckle tracking algorithm, which uses block matching and normalized cross-correlation function to track the displacement of tissue speckle over time.

[0054] In another possible implementation, a YOLO target detection algorithm is selected to process the ultrasonic echo signal, and the version of the YOLO is not less than YOLOv5.

[0055] In another possible implementation, a cross-correlation analysis algorithm is selected to process the ultrasonic echo signal, which calculates the cross-power spectrum of adjacent frame complex signals in the frequency domain and inversely transforms to obtain a displacement field.

[0056] In another possible implementation, Kalman filtering is adopted to process the magnetic induction signal to estimate the signal and noise simultaneously in a state space model, thereby improving dynamic accuracy.

[0057] In another possible implementation, the optical flow method is used to process the ultrasound echo signal, and the Lucas-Kanade sparse optical flow is used to track the displacement of tissue feature points, so as to reduce the calculation amount of spot tracking.

[0058] In another possible implementation, a deep convolutional neural network regression is used to process the ultrasound echo signal, and a displacement vector is directly output, the network takes a B-mode image sequence as input, and is trained in an end-to-end manner without explicit spot assumption.

[0059] 103、Information fusion: using a weighted fusion algorithm, the body surface displacement information and the organ tissue displacement information are extracted variance reciprocal, normalized signal-to-noise ratio and normalized cross-correlation coefficient, the normalized signal-to-noise ratio and the normalized cross-correlation coefficient are multiplied after sigmoid mapping as the credibility, the credibility and the variance reciprocal are point multiplied and normalized to obtain the weight, the body surface displacement information and the organ tissue displacement information are weighted and summed according to the weight, to obtain weighted fusion data; the body surface displacement information and the organ tissue displacement information are Kalman filter fused to obtain Kalman fusion data; the weighted fusion data and the Kalman fusion data are combined to obtain fusion displacement data.

[0060] In this embodiment, the Kalman filter takes the body surface displacement information as an observation vector and the organ tissue displacement information as a state vector, and estimates the optimal displacement through a prediction-update cycle.

[0061] In this embodiment, the combination of the weighted fusion data and the Kalman fusion data adopts Bayesian model averaging, and the weighted fusion data and the Kalman fusion data are used as candidate models, and are weighted and summed according to posterior probability.

[0062] In another possible implementation, the information fusion adopts a particle filter framework, takes the body surface displacement as an observation and the organ displacement as a hidden state, and realizes robust estimation under non-Gaussian noise through resampling and weight updating.

[0063] In another possible implementation, the information fusion adopts adaptive weighted average, and the weight is dynamically adjusted through mutual information of the body surface displacement information and the organ tissue displacement information estimated in real time, so as to ensure that a high-confidence channel occupies a larger proportion.

[0064] In another possible implementation, the combination of the weighted fusion data and the Kalman fusion data adopts other combination methods, such as weighted average, covariance cross filtering, stacked generalization, etc.

[0065] 104、Mapping establishment: using a bidirectional LSTM network, a respiratory phase-organ displacement mapping relationship is constructed according to the fusion displacement data.

[0066] In the embodiment, the bidirectional LSTM network is formed by a forward layer and a backward layer in parallel, the forward layer reads the fused displacement data in time increasing order, the backward layer reads the fused displacement data in time decreasing order, and the two layers of hidden states are spliced through a full connection layer to output an organ displacement curve corresponding to a respiratory phase; the respiratory phase is extracted from the fused displacement data by Hilbert transform to obtain a sequence of instantaneous phase angles, the phase angles are normalized to serve as network input, and a weighted mean square error is used as the network training loss function, different weights are respectively assigned to the expiration phase and the inspiration phase to highlight the features of the sudden change section.

[0067] In another possible implementation, other network training loss functions are used.

[0068] In another possible implementation, the construction of the respiratory phase-organ displacement mapping relationship is implemented using a transformer-based network.

[0069] In another possible implementation, the construction of the respiratory phase-organ displacement mapping relationship uses a time convolution network (TCN), and the dilated convolution structure of the TCN can capture multi-scale respiratory cycle features while maintaining the advantage of parallel computing.

[0070] In another possible implementation, the construction of the respiratory phase-organ displacement mapping relationship uses a Gaussian process regression, takes the respiratory phase as input and the displacement as output, learns a nonlinear mapping through a kernel function, and outputs an uncertainty estimate.

[0071] 105、Displacement prediction: predicting the organ displacement amount at a future time according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement amount.

[0072] In the embodiment, the prediction is implemented in a combination of an LSTM network and a linear harmonic respiratory physical equation, and the LSTM network outputs a preliminary displacement sequence.

[0073] In the embodiment, the linear harmonic respiratory physical equation uses a sinusoidal harmonic superposition chest pressure rate of change model to physically constrain and correct the sequence, and the formula is as follows:

[0074] ,

[0075] wherein is the displacement amount of the organ, t is a time variable, f is a respiratory frequency, is a chest pressure rate of change, a is a displacement amplitude, b represents the sensitivity of the organ displacement to the pressure rate of change, is a phase offset, and finally the organ displacement amount at the future time is obtained, and a displacement compensation instruction is triggered to a surgical robot or a treatment bed.

[0076] In another possible implementation, the prediction adopts an extended Kalman filter to fuse the LSTM network output and a respiratory physical equation, and uses a nonlinear state transition model to correct a prediction error in real time.

[0077] In another possible implementation, the prediction adopts a model predictive control framework, uses the LSTM network output as a system model, rolls an optimal displacement trajectory in a plurality of future time windows, and issues an optimal control amount as a displacement compensation instruction.

[0078] In some embodiments, the apparatus provided by the embodiments of the present application has functions or includes modules that can be used to perform the methods described in the above method embodiment, and the specific implementation can refer to the description of the above method embodiments. For brevity, it will not be repeated here.

[0079] The above describes the method of the embodiments of the present application in detail. The apparatus of the embodiments of the present application is provided below.

[0080] Please refer to Figure 4 , Figure 4 A schematic diagram of an intraoperative respiratory follow-up device based on magnetic-ultrasound signals is provided for the embodiments of the present application. The respiratory follow-up device 1 includes a sensing unit 11, a preprocessing unit 12, a fusion unit 13, a mapping unit 14, and a prediction unit 15, specifically:

[0081] The sensing unit 11 includes: a. a body surface ultrasound probe, configured to acquire an ultrasound echo signal intraoperatively;

[0082] b. a wearable magnetic sensor array, configured to acquire a magnetic induction signal intraoperatively, for details, please refer to Figure 2 、 Figure 3 ;

[0083] C. a time synchronizer, configured to synchronize the operation of the body surface ultrasound probe and the wearable magnetic sensor array;

[0084] The preprocessing unit 12 is configured to process the magnetic induction signal to obtain body surface displacement information, and process the ultrasound echo signal to obtain organ tissue displacement information;

[0085] The fusion unit 13 is configured to use a Kalman filter and a weighted fusion algorithm to fuse the body surface displacement information and the organ tissue displacement information to obtain fused displacement data;

[0086] The mapping unit 14 is configured to use a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fused displacement data;

[0087] The prediction unit 15 is configured to predict an organ displacement amount at a future time according to the mapping relationship, and send a displacement compensation instruction based on the organ displacement amount.

[0088] In some embodiments, the apparatus provided by the embodiments of the present application has functions or includes modules for performing the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For briefness, it will not be described here in detail.

[0089] Please refer to Figure 5 , Figure 5 A hardware architecture schematic diagram of an electronic device is shown in the embodiments of the present application. The electronic device 2 is mainly composed of a processor 21 and a memory 22. In addition, the device can also include an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are connected to each other through connection components, which can be various interfaces, data lines or communication buses, etc. The embodiments of the present application do not make specific provisions for this.

[0090] The processor 21 can be one or more graphic processors (GPU). If the processor 21 is a GPU, the GPU can be single-core or multi-core. As an option, the processor 21 can also be composed of a processor group composed of multiple GPUs, and these processors are connected to each other through one or more buses. In addition, the processor can also be other types of processors, and the embodiments of the present application do not make specific limitations.

[0091] The memory 22 is designed to save the instructions of the computer program and various program codes required for the implementation of the scheme of the present application. As an option, the memory can include but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or compact disc read-only memory (CD-ROM), which are used to store related instructions and data.

[0092] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices, or they can be an integral device.

[0093] It should be recognized that, in the embodiments of the present application, the memory 22 can not only save related instructions, but also save related data. The embodiments of the present application do not make specific provisions for the specific data content stored in the memory.

[0094] It should be understood that Figure 5 Only a simplified design of an electronic device is shown. In actual use, the electronic device can also include other necessary components, such as different numbers of input / output devices, processors, memories, etc. All electronic devices capable of implementing the embodiments of the present application belong to the protection scope of the present application.

[0095] Those skilled in the art should understand that, according to the components and algorithm steps of various examples described in the embodiments disclosed herein, the functions can be realized by electronic hardware or in combination with computer software and electronic hardware. Whether the functions are performed by hardware or software depends on the specific application and design constraints of the technical solutions. Those skilled in the art can adopt different implementation methods for each specific application, but the implementation methods should not be considered as beyond the protection scope of the present application.

[0096] It should be understood by those skilled in the art that, for the convenience of description and simplification of the description, the specific operation processes of the above-mentioned system, device and component can refer to the corresponding steps in the foregoing method embodiments, which will not be repeated here. Meanwhile, those skilled in the art should understand that each embodiment in the present application has its own emphasis, and for the convenience of description and simplification, the same or similar contents may not be repeatedly described in different embodiments, so if the part is not mentioned or not described in detail in an embodiment, it can be referred to the relevant description of other embodiments.

[0097] In several embodiments provided in the present application, it should be recognized that the disclosed system, device and method can also be realized by other ways. For example, the described device embodiments are only exemplary, and the division of the units is only logical functional division, and different division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or some steps may not be performed. In addition, the connection between the units shown or discussed, whether direct or indirect, whether coupled or communicatively connected, can be realized through interfaces, devices or units in electrical, mechanical or other forms.

[0098] The units described as independent components may actually be physically separated or not; the parts presented as units may or may not be physical entities, that is, they can be concentrated in one location or dispersed on multiple network nodes. According to the actual needs, part or all of these units can be selected to achieve the goal of the present embodiment.

[0099] Moreover, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each independent physical unit, or two or more units can be combined into one unit. In the foregoing embodiments, the related functions can be fully or partially implemented by software, hardware, firmware or any combination thereof. If software implementation is chosen, it can be implemented in whole or in part in the form of a computer program product. The computer program product contains one or more computer instructions. When the instructions are loaded and executed on a computer, they produce all or part of the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, DSL) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server, data center, etc. integrated with one or more available media. These available media can include magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art should understand that all or part of the processes of the above-mentioned embodiments can be completed by computer program instruction related hardware, and these programs can be stored in a computer-readable storage medium. When these programs are executed, they will contain the processes of the above-mentioned embodiments. The above-mentioned storage medium includes but is not limited to read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various media that can store program codes.

Claims

1. An intraoperative respiratory following method based on magnetic-ultrasound signals, characterized in that, The method comprises the following steps: a. synchronously acquiring magnetic induction signals and ultrasonic echo signals during surgery; b. processing the magnetic induction signals to obtain body surface displacement information, and processing the ultrasonic echo signals to obtain organ tissue displacement information; c. using a weighted fusion algorithm, extracting the variance inverse, normalizing the signal-to-noise ratio and the normalized cross-correlation coefficient of the body surface displacement information and the organ tissue displacement information, multiplying the normalized signal-to-noise ratio and the normalized cross-correlation coefficient after sigmoid mapping as the reliability, point-multiplying the reliability and the variance inverse to obtain the weight, and normalizing the weight to obtain the weighted sum of the body surface displacement information and the organ tissue displacement information, obtaining the weighted fusion data; Kalman filtering fusion is performed on the body surface displacement information and the organ tissue displacement information to obtain Kalman fusion data; the weighted fusion data and the Kalman fusion data are combined to obtain fusion displacement data; d. using a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fusion displacement data; e. predicting the organ displacement amount at a future time according to the mapping relationship, and sending a displacement compensation instruction based on the organ displacement amount.

2. The method of claim 1, wherein, The synchronous acquisition is based on a hardware trigger signal or a unified timestamp.

3. The method of claim 1, wherein, The processing of the magnetic induction signals includes filtering and solving the magnetic induction signals; and the processing of the ultrasonic echo signals includes speckle tracking or cross-correlation analysis of the ultrasonic echo signals.

4. The method of claim 1, wherein, The Kalman filter takes the body surface displacement information as an observation vector and the organ tissue displacement information as a state vector, and estimates the optimal displacement through a prediction-update cycle; and the combination of the weighted fusion data and the Kalman fusion data adopts a Bayesian model average, takes the weighted fusion data and the Kalman fusion data as candidate models, and performs weighted summation according to posterior probability.

5. The method of claim 1, wherein, The bidirectional LSTM network comprises a forward layer and a backward layer connected in parallel, the forward layer reads the fusion displacement data in time increasing order, the backward layer reads the fusion displacement data in time decreasing order, the hidden state is spliced through a full connection layer and then output, and an organ displacement curve corresponding to a respiratory phase is output; the respiratory phase is extracted from the fusion displacement data through Hilbert transform to obtain an instantaneous phase angle sequence, and the phase angle is normalized and taken as network input.

6. The method of claim 1, wherein, The prediction of the organ displacement amount at the future time is performed in a manner that combines a long short-term memory network model with a respiratory physical equation; the respiratory physical equation is based on a linear harmonic model, and the formula is: , wherein is the displacement of the organ, t is the time variable, f is the respiratory frequency, is the rate of change of the thoracic pressure, a is the displacement amplitude, b represents the sensitivity of the organ displacement to the rate of change of the pressure, is the phase shift.

7. An intraoperative respiratory following device based on magnetic-ultrasound signals, characterized in that, It comprises: a sensing unit comprising: a. a body surface ultrasonic probe for acquiring ultrasonic echo signals during surgery; b. a wearable magnetic sensor array for acquiring magnetic induction signals during surgery; and c. a time synchronizer for synchronizing the operation of the body surface ultrasonic probe and the wearable magnetic sensor array; a preprocessing unit for processing the magnetic induction signals to obtain body surface displacement information, and processing the ultrasonic echo signals to obtain organ tissue displacement information; ​ ​ The fusion unit is configured to use a weighted fusion algorithm to extract a variance reciprocal, a normalized signal-to-noise ratio and a normalized cross-correlation coefficient of the body surface displacement information and the organ tissue displacement information, multiply the normalized signal-to-noise ratio and the normalized cross-correlation coefficient after sigmoid mapping as a credibility, multiply the credibility and the variance reciprocal to obtain a weight, perform weighted summation on the body surface displacement information and the organ tissue displacement information according to the weight, obtain weighted fusion data, perform Kalman filtering fusion on the body surface displacement information and the organ tissue displacement information to obtain Kalman fusion data, and combine the weighted fusion data and the Kalman fusion data to obtain fused displacement data. The mapping unit is configured to use a bidirectional LSTM network to construct a respiratory phase-organ displacement mapping relationship according to the fused displacement data. The prediction unit is configured to predict an organ displacement amount at a future time according to the mapping relationship, and send a displacement compensation instruction based on the organ displacement amount.

8. An electronic device, comprising: The electronic device comprises a processor and a storage unit configured to store computer program code, the code comprising computer instructions, when the processor executes the instructions, the electronic device executes the method of any one of claims 1 to 6. The computer readable storage medium stores a computer program, the computer program contains program instructions, and when the program instructions are executed by the processor, the processor executes the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program product contains computer programs or instructions, when the computer programs or instructions run on the computer, so that the computer executes the method of any one of claims 1 to 6.

10. A computer program product, characterised in that, ​

Images