Method and apparatus for predicting visceral movement, and electronic device

Abstract

The application discloses a method and device for predicting visceral movement, and an electronic device. The method comprises: acquiring magnetic induction data of the external skin of the internal organs at the current time, wherein the magnetic induction data comprises movement information of the external skin at different coordinates; inputting the magnetic induction data into a trained prediction model, and outputting predicted magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time; inputting the predicted magnetic induction data into a trained fitting model, and outputting movement data of the internal organs from the fitting model. The method solves the problem in the related art that, when the internal organs are subjected to ultrasonic imaging, the position of the internal organs changes due to the movement of the measured person, such as breathing, so that the imaged position of the internal organs is the position of the internal organs before the current time, and the movement of the internal organs cannot be learned in time.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical assistance, in particular to a method and device for predicting internal organ movement. BACKGROUND

[0002] Currently, in internal organ surgery, the position of the internal organ can be determined through ultrasound imaging of the internal organ, and then surgery assistance can be performed. However, the position of the internal organ changes with breathing and movement, and ultrasound imaging has a certain time lag. Therefore, the latest ultrasound imaging obtained is not the current position and state of the internal organ, but the previous position and state of the internal organ. Thus, surgery assistance at the current time is difficult to achieve accurate and effective results. According to the current prior art, the accurate position of the internal organ cannot be obtained in time.

[0003] In the related art, when the internal organ is imaged, the position of the internal organ changes due to the breathing and movement of the subject, resulting in the imaged position of the internal organ being the position of the internal organ at a time before the current time, and the movement of the internal organ cannot be obtained in time. An effective solution has not yet been proposed. SUMMARY

[0004] The main purpose of the present application is to provide a method and device for predicting internal organ movement, to solve the problem that in the related art, when the internal organ is imaged, the position of the internal organ changes due to the breathing and movement of the subject, resulting in the imaged position of the internal organ being the position of the internal organ at a time before the current time, and the movement of the internal organ cannot be obtained in time.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present application, a method for predicting internal organ movement is provided, the method comprising: acquiring magnetic induction data of the external skin of the internal organ at the current time, wherein the magnetic induction data comprises movement information of the external skin at different coordinates; inputting the magnetic induction data into a trained prediction model, outputting predicted magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time; inputting the predicted magnetic induction data into a trained fitting model, and outputting movement data of the internal organ from the fitting model.

[0006] Optionally, before the magnetic induction data is input into the trained prediction model to output the predicted magnetic induction data at the target time, the method further comprises: obtaining first historical data of the magnetic induction data, wherein the first historical data is magnetic induction data before the current time; extracting a plurality of groups of first training data from the first historical data, wherein each group of first training data comprises magnetic induction data at a previous time as model input and magnetic induction data at a subsequent time as model output, and the time interval between the previous time and the subsequent time is the same as the time interval between the current time and the target time; training the prediction model according to the plurality of groups of first training data until the prediction error of the prediction model reaches a first preset requirement.

[0007] Optionally, after the magnetic induction data is input into the trained prediction model to output the predicted magnetic induction data at the target time, the method further comprises: collecting target magnetic induction data at the target time; updating the first historical data according to the target magnetic induction data to update the corresponding training data; and updating the prediction model according to the updated training data to obtain an updated prediction model.

[0008] Optionally, training the prediction model according to the plurality of groups of first training data until the prediction error of the prediction model reaches a first preset requirement comprises: creating an initial model according to a multi-layer perception network, wherein the multi-layer perception network comprises a plurality of network parameters, and the plurality of network parameters at least comprise at least one of the following: a prediction window input number, a hidden layer number, and the multi-layer perception network learns network parameters through a Bayesian optimization algorithm; initializing the initial model, and performing supervised learning training on the initial model through the plurality of groups of first training data using a normal propagation algorithm; and in a case where it is detected that the prediction error of the initial model satisfies the first preset requirement, taking the trained initial model as the prediction model, wherein the prediction error comprises a maximum error ME and a root mean square error RMSE.

[0009] Optionally, obtaining the magnetic induction data of the internal organs and the external skin at the current time comprises: collecting raw magnetic induction data through an EM sensor arranged on the external skin; converting the raw magnetic induction data into a one-dimensional signal in a three-dimensional coordinate direction, and determining a first main direction for marking the respiratory motion of the external skin according to the one-dimensional signal; performing data normalization, data smoothing, and abnormal value removal processing on the first main direction to obtain the magnetic induction data.

[0010] Optionally, before the prediction magnetic induction data is input into the trained fitting model, the method further comprises: acquiring second historical magnetic induction data, wherein the second historical magnetic induction data comprises magnetic induction data and ultrasound data synchronously collected in a historical time period; extracting a plurality of second training data from the second historical magnetic induction data, wherein each set of second training data comprises magnetic induction data and ultrasound data at the same time, the magnetic induction data serving as input and the ultrasound data serving as output; and training the fitting model according to the plurality of second training data until a fitting error of the fitting model reaches a second preset requirement.

[0011] Optionally, acquiring the second historical magnetic induction data comprises: acquiring ultrasound data of the internal organ collected historically, and magnetic induction data of the external skin collected historically; wherein the ultrasound data of the internal organ collected historically is used to collect ultrasound images of the internal organ; registering two adjacent ultrasound images to obtain offset information of deformation field pixels; calculating offset motion information according to the offset information; and performing data normalization, data smoothing and abnormal value removal processing on the offset motion information to obtain the ultrasound data.

[0012] Optionally, training the fitting model according to the plurality of second training data until the fitting error of the fitting model reaches the second preset requirement comprises: creating a motion model by a multivariate linear regression method, wherein the motion model is as follows: wherein Y(t) represents ultrasound data of internal organ motion, M(t) i represents the i-th magnetic induction data parameter, n represents the number of magnetic induction data parameters, M(t) 0 is a virtual unit intercept used to represent a bias of the motion model, and β i is a coefficient of each magnetic induction data parameter; training the fitting model according to the plurality of second training data to determine the fitting error, wherein the fitting error comprises a root mean square error (RMSE); and in a case where the fitting error reaches the second preset requirement, taking the trained motion model as the fitting model.

[0013] To achieve the above object, according to another aspect of the present application, a prediction device for internal organ motion is provided, comprising: an acquisition module configured to acquire magnetic induction data of an external skin of an internal organ at a current time, wherein the magnetic induction data comprises motion information of the external skin at different coordinates; a prediction module configured to input the magnetic induction data into a trained prediction model, and output prediction magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time; and a fitting module configured to input the prediction magnetic induction data into a trained fitting model, and output motion data of the internal organ from the fitting model.

[0014] In order to achieve the above object, according to another aspect of the present application, a computer readable storage medium is provided for storing a program, wherein the program implements the visceral motion prediction method according to any one of the above.

[0015] In order to achieve the above object, according to another aspect of the present application, an electronic device is provided, comprising one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the visceral motion prediction method according to any one of the above.

[0016] The present application collects the magnetic induction data of the skin outside the internal organs, predicts the magnetic induction data at the target time through the prediction model, and then fits the visceral motion data corresponding to the magnetic induction data at the target time through the fitting model according to the predicted magnetic induction data at the target time. The purpose of determining the visceral motion at the target time through the collection of the magnetic induction data of the skin outside the internal organs at the current time is achieved, the technical effect of knowing the visceral motion at the target time in advance is realized, and the problem that the internal organs are imaged by ultrasound, the position of the internal organs changes due to the movement of the measured person, such as breathing, the imaged internal organs are the position of the internal organs before the current time, and the visceral motion cannot be known in time in the related art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety, and the illustrative embodiments of the present application and their description serve to explain the present application. In the drawings:

[0018] Figure 1 is a flowchart of a visceral motion prediction method according to an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the overall architecture of the visceral motion prediction flow according to the embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the magnetic induction data collection method according to the embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the ultrasound data collection method according to the embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the division of training data, validation data and test data of the prediction model according to the embodiment of the present application;

[0023] Figure 6is a schematic diagram of a sliding window method principle provided according to an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of a visceral movement prediction device provided according to an embodiment of the present application;

[0025] Figure 8 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0029] The present application will be described below in combination with preferred implementation steps, Figure 1 is a flowchart of a visceral movement prediction method provided according to an embodiment of the present application, as shown in Figure 1 The method comprises the following steps:

[0030] Step S101, acquiring magnetic induction data of the external skin of the viscera at the current time, wherein the magnetic induction data comprises movement information of the external skin at different coordinates;

[0031] Step S102, inputting the magnetic induction data into the trained prediction model, and outputting predicted magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time;

[0032] Step S103, input the predicted magnetic induction data into the trained fitting model, and output the motion data of the internal organs from the fitting model.

[0033] The above steps achieve the purpose of determining the motion of the internal organs at the target time through the collection of the magnetic induction data of the external skin of the internal organs, and realizing the technical effect of knowing the motion of the internal organs at the target time in advance, thereby solving the problem in the related art that the position of the internal organs changes due to the motion such as breathing of the measured person when the internal organs are imaged by ultrasound, resulting in that the imaged position of the internal organs is the position of the internal organs before the current time, and the motion of the internal organs cannot be known in time.

[0034] The execution subject of the above steps can be an internal organ motion prediction system, which can include a processing device to perform the data processing operations in the above steps, such as steps S101-S103. The prediction system can also include a sensing device for collecting magnetic induction data, and an ultrasound device for collecting ultrasound data. It should be noted that the magnetic induction data collected by the sensing device is the data of the external skin, and the ultrasound data collected by the ultrasound device is the actual data of the internal organs, and the sensing device and the ultrasound device synchronously collect the magnetic induction data and the ultrasound data for training the fitting model.

[0035] Specifically, the above magnetic induction data can include coordinate signals of the external skin, which are collected by the EM sensor attached to the external skin. The above ultrasound data can include displacement images of the internal organs, which are obtained by ultrasonic detection of the internal organs by an ultrasonic probe of an ultrasonic imaging device.

[0036] When collecting the magnetic induction data, it is continuously collected in time, so that continuous magnetic induction data within a certain time period is obtained. A fixed time window, for example, 2s, is used to intercept the magnetic induction data to obtain a group of data, including the magnetic induction data at the previous time and the magnetic induction data at the subsequent time 2s after the previous time. Through the movement of the time window, multiple groups of data can be obtained as training data to train the above prediction model. The above prediction model is used to predict the magnetic induction data at the subsequent time according to the input magnetic induction data at the previous time.

[0037] When the above steps are executed, the prediction model and the fitting model have been trained and are directly used.

[0038] In step S101, the magnetic induction data of the internal organs at the current time can be obtained in the same way as described above. For example, the magnetic induction data is collected by the EM sensor attached to the external skin. Then, the magnetic induction data is input into the prediction model, and the magnetic induction data at the target time is output by the prediction model. Then, the magnetic induction data at the target time is input into the fitting model, and the internal organ movement data corresponding to the magnetic induction data is output by the fitting model. Furthermore, the internal organ movement data can be obtained or known before the target time arrives.

[0039] Optionally, before the magnetic induction data is input into the trained prediction model to output the predicted magnetic induction data at the target time, the method further comprises: obtaining first historical data of the magnetic induction data, wherein the first historical data is the magnetic induction data before the current time; extracting a plurality of first training data from the first historical data, wherein each set of first training data includes the magnetic induction data at the previous time as the model input and the magnetic induction data at the subsequent time as the model output, and the time interval between the previous time and the subsequent time is the same as the time interval between the current time and the target time; and training the prediction model according to the plurality of first training data until the prediction error of the prediction model meets the first preset requirement.

[0040] Before the prediction model is used, the latest magnetic induction data before the current time can be obtained to update the first historical data, and then the first training data is updated to train the prediction model, so that the prediction model has better and more stable dynamic ability as the use time increases. The first historical data can be the magnetic induction data within a certain time window before the current time. The plurality of first training data can be extracted from the first historical data by the time window.

[0041] It should be noted that the prediction error of the prediction model can be represented by the maximum error ME and the root mean square error RMSE, that is, the first preset requirement that the prediction error of the prediction model meets can be that the maximum error ME and the root mean square error RMSE of the prediction model meet the preset requirement.

[0042] Optionally, after the magnetic induction data is input into the trained prediction model to output the predicted magnetic induction data at the target time, the method further comprises: collecting target magnetic induction data at the target time; updating the first historical data according to the target magnetic induction data to update the corresponding training data; and updating the training data to obtain an updated prediction model.

[0043] After the target time arrives, the target magnetic induction data at the target time is collected to update the first historical data, and then the first training data is updated to dynamically train and update the prediction model, so that the prediction model has good prediction ability.

[0044] Optionally, training the prediction model based on multiple sets of first training data until the prediction error of the prediction model reaches a first preset requirement includes: creating an initial model based on a multilayer perceptron network, wherein the multilayer perceptron network includes multiple network parameters, and the multiple network parameters include at least one of the following: the number of prediction window inputs, the number of hidden layers, and the multilayer perceptron network learns the network parameters through a Bayesian optimization algorithm; initializing the initial model by performing supervised learning training on the initial model using a normal propagation algorithm based on multiple sets of first training data; and using the trained initial model as the prediction model when the prediction error of the initial model meets the first preset requirement, wherein the prediction error includes the maximum error ME and the root mean square error RMSE.

[0045] The prediction model in this embodiment is built using a multilayer perceptron network with adaptive parameter learning. Network parameters, such as the number of input windows and hidden layers, can be learned through a Bayesian optimization algorithm. A backpropagation (BP) algorithm with adaptive momentum is chosen to train the network. The system is initialized with random weights and bias values ​​to reduce the risk of symmetric initial conditions. The training phase in supervised learning requires simulating the input pattern and comparing it with the desired output signal. Preferably, the Levenberg-Marquadt optimization algorithm can be selected, which is a fast and commonly used BP method in nonlinear optimization.

[0046] Optionally, acquiring the magnetic sensing data of the external skin of the internal organs at the current moment includes: collecting raw magnetic sensing data through an EM sensor set on the external skin; converting the raw magnetic sensing data into a one-dimensional signal in the three-dimensional coordinate direction, and determining the first principal direction for marking the respiratory motion of the external skin based on the one-dimensional signal; and performing data normalization, data smoothing, and outlier removal on the first principal direction to obtain the magnetic sensing data.

[0047] For magnetic field data with three degrees of freedom in the X, Y, and Z directions, the three-dimensional motion of the external marker is converted into a one-dimensional signal by projecting three-dimensional coordinates into the principal component space. Principal component analysis (PCA) is applied to the marker trajectory obtained during the training phase to extract the first principal direction of the marker motion. The external marker motion at each time point t is then analyzed. It is expressed as follows: Where P is a mathematical operator that projects 3D coordinates onto the first principal direction.

[0048] Movement using external markers This is used to characterize the movement of the external skin corresponding to the magnetic field data. Data normalization, smoothing, and outlier removal can make the magnetic field data more accurate.

[0049] Optionally, before inputting the predicted magnetic field data into the trained fitting model and having the fitting model output the visceral motion data, the method further includes: acquiring second historical data of the magnetic field data, wherein the second historical data consists of magnetic field data and ultrasound data collected synchronously within a historical time period; extracting multiple sets of second training data from the second historical data, wherein each set of second training data includes magnetic field data and ultrasound data at the same moment, with the magnetic field data as input and the ultrasound data as output; training the fitting model based on the multiple sets of second training data until the fitting error of the fitting model reaches a second preset requirement.

[0050] When training the fitting model, synchronously acquired magnetic field data and ultrasound data are used as a second set of training data. The magnetic field data and ultrasound data in the same second set of training data represent data from the external skin and internal organs acquired at the same time. By training the fitting model using this second set of training data, the model can generate visceral data from the magnetic field data at the same time, thereby determining the movement of the internal organs.

[0051] It should be noted that the fitting error of the above fitting model can be the root mean square error (RMSE). That is, the second preset requirement that the fitting error of the fitting model meets is that the root mean square error (RMSE) of the fitting model meets the preset requirement.

[0052] Optionally, the second historical data for acquiring magnetic sensing data includes: acquiring historically acquired ultrasound data of internal organs and historically acquired magnetic sensing data of external skin; wherein, during the acquisition of historically acquired ultrasound data of internal organs, ultrasound images of the internal organs are acquired; two adjacent frames of ultrasound images are registered to obtain the offset information of the deformation field pixels; offset motion information is calculated based on the offset information; and the offset motion information is processed by data normalization, data smoothing, and outlier removal to obtain ultrasound data.

[0053] The ultrasound data can be 2D ultrasound images. B-spline non-rigid registration is performed on these 2D ultrasound images, employing multi-metric, multi-resolution registration. The optimization method uses adaptive stochastic gradient descent with 1000 iterations. Registration is performed between adjacent frames. The pixel offset information of the resulting deformation field after registration is accumulated and averaged to obtain the average displacement motion of the internal tumor.

[0054] Through average offset motion This is used to characterize the movement of internal organs. Data normalization, smoothing, and outlier removal can make ultrasound data more accurate.

[0055] Optionally, the fitting model is trained according to multiple sets of second training data until a fitting error of the fitting model reaches a second preset requirement, including: the fitting model needs to be created according to a multiple linear regression method before training, where the motion model is as follows: where Y(t) is ultrasound data representing visceral motion, M(t) i represents the i-th magnetic induction data parameter, n represents the number of magnetic induction data parameters, M(t)0 is a virtual unit intercept for representing the bias of the motion model, and β i is a coefficient of each magnetic induction data parameter; the fitting model is trained according to multiple sets of second training data to determine a fitting error, where the fitting error includes a root mean square error RMSE; and in a case where the fitting error reaches a second preset requirement, the trained motion model is taken as the fitting model.

[0056] The motion model is used to fit the external marker motion and the average offset motion , and has better performance and effects. The motion model can be used to accurately and quickly fit the external marker motion to obtain the average offset motion

[0057] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that herein.

[0058] It should be noted that the present application also provides an optional embodiment, which will be described in detail below.

[0059] The present embodiment provides a liver motion prediction model based on a skin-mounted position sensor and a corresponding sequence of ultrasound liver images, as shown in Figure 2 , and Figure 2 is a schematic diagram of the overall architecture of the visceral motion prediction process provided by the embodiment of the present application. The technical solution of the present embodiment will be described in detail below.

[0060] I. Data set acquisition: magnetic induction data (external skin coordinate signal External markers position): Dm; ultrasound data (internal liver displacement image Internal markers position): Dus.

[0061] Step 1: The ultrasonic data Dus is collected by using a 2D ultrasonic probe with a specification of 7.5Mhz and 40mm to perform a B-mode ultrasonic image of the liver sagittal plane. The time resolution is 10Hz, the image pixel size is approximately 960*720mm, and the collection time is approximately 3 minutes.

[0062] The magnetic induction data Dm is collected by using an electromagnetic position tracker EM for the displacement of the umbilical skin surface. The collection frequency is 10Hz, and two six-degree-of-freedom displacement sensors are externally connected. The collection time is approximately 3 minutes.

[0063] Step 2: The subject lies flat on the recliner, adopts the supine position, raises both hands behind the head, and performs free and stable breathing. The experimenter attaches two EM sensors to the umbilicus on the abdominal surface of the subject, as Figure 3 shown, Figure 3 is a schematic diagram of the magnetic induction data collection method provided according to the implementation manner of the present application. The ultrasonic probe (with the head facing up) is fixed by hand on the liver side of the subject, as Figure 4 shown, Figure 4 is a schematic diagram of the ultrasonic data collection method provided according to the implementation manner of the present application.

[0064] II. Dm data preprocessing.

[0065] Step I: For the data Dm with three degrees of freedom in the X, Y, and Z directions, the three-dimensional motion of the external marker is converted into a one-dimensional signal by projecting the three-dimensional coordinates in the principal component space. Principal component analysis (PCA) is applied to the marker trajectories obtained in the training stage to extract the first principal direction of the marker motion. The external marker motion at each time point t is expressed as follows: where P is a mathematical operator that projects the 3D coordinates onto the first principal direction.

[0066] Step II: The dimensionality-reduced surrogate data is subjected to data normalization, data smoothing, and outlier removal to obtain the external marker motion at each time point t where V represents the mathematical operator for the three data processes.

[0067] Data smoothing: Fourier smoothing is adopted.

[0068] Outlier removal: Remove outliers that exceed 2 / 3 of the maximum amplitude of the average breathing cycle.

[0069] Data normalization: The surrogate data in absolute coordinates is converted into the relative offset of the breathing displacement.

[0070] III. Dus imaging data processing.

[0071] Step one: B-spline non-rigid registration is performed on the 2D ultrasound images, in which multi-measure index multi-resolution is used for registration, and adaptive stochastic gradient descent and gradient descent are used for optimization, with 1000 iterations. The registration method is to register adjacent two frames.

[0072] Step two: the pixel point offset information generated by the registration is accumulated and averaged to obtain the average offset motion of the internal tumor

[0073] Step three: the internal tumor offset motion is normalized, smoothed and outliers are removed. Repeat step two of the magnetic sensing data. Obtain the internal tumor motion at each time point t where V represents a mathematical operator for three data processing.

[0074] Four: Dus-Dm correlation modeling model.

[0075] Step one: the liver motion is represented by the Dus average displacement Y(t) of the liver region of interest, and the abdominal motion is represented by the Dm tracking displacement M(t) of the marker.

[0076] Step two: based on the model in this embodiment, the model is specific to the patient and the course of treatment, so a linear representation of the fitting algorithm is sufficient. Therefore, a linear fitting method is selected in this paper, in which the most commonly used method is the multiple linear regression method (MVR), therefore, the motion model formula is as follows: where Y(t) represents the internal liver motion, β = [β0, β1……, β n ] represents the parameters / coefficients of the motion model, M I = [M t1 , M t2 , …, M tn ] represents that the model features are determined from the alternative data M(t), n represents the number of parameters, and M(t)0 represents a virtual unit intercept for explaining the bias in the model.

[0077] Step three: Figure 5 is a schematic diagram of the division of training data, validation data and test data of the prediction model provided according to the embodiments of the present application, as shown in Figure 5 The obtained data set is divided into three parts: training data, validation data and test data, the training data is used to fit the model, the validation data is used to select features and shrinkage parameters, and the test data is used to evaluate the performance of the model. The proportions of test, validation and training data are 50%, 20% and 30% respectively.

[0078] Step four: After fitting the ultrasound data Y(t) and the magnetic sensor data M(t), the surrogate data M(t) is input into the correlation model (MVR) for calculation when verifying and testing the fitting effect Output the fitted ultrasound data

[0079] Step five: The error of the correlation model fitting is evaluated by the root mean square error (RMSE), which represents the sample standard deviation of the difference between the predicted value and the true value.

[0080] Five, respiratory phase prediction.

[0081] According to the correlation model The internal liver motion Y(t) is replaced by external data M(t). In order to make up for the system data transmission delay and the requirements of clinical surgery, a certain time delay is required for the liver motion position, and the artificial neural network has good regression prediction performance for time series data. The prediction algorithm is processed in MATLAB2018a.

[0082] Step one: This embodiment adopts a multi-layer perception network with adaptive parameter learning, and a Bayesian optimization algorithm is used to learn the network parameters, such as the number of input windows, the number of hidden layers, etc.

[0083] And a back propagation (BP) algorithm with adaptive momentum is selected to train the network. The system is initialized with random weight and bias values, aiming to reduce the risk of symmetric initial conditions. The training phase in supervised learning requires simulating input patterns and comparing them with the expected output signal. In this embodiment, the Levenberg-Marquadt optimization algorithm is selected, which is a fast and commonly used BP method in nonlinear optimization.

[0084] Step two: Since the respiratory time series is naturally non-stationary, it has quasi-periodic frequency, mean value and amplitude that change over time, so it is impossible to obtain a universally valid model from a single training data set. Therefore, the influence of real-time retraining of the prediction model on its prediction accuracy is studied. By retraining with the latest history of measured values, the contemporary effective control law of the non-stationary system is captured. The model is retrained before each new measurement sample, i.e. before each new sample prediction.

[0085] This method can be called a sliding window method, Figure 6 is a schematic diagram of the principle of the sliding window method provided by the embodiment of the present application, as Figure 6 shown, the sliding window of the time series data obtained from the respiratory cycle is input into the neural network to predict the future position. The model is trained using a limited past history of respiratory position.

[0086] Step three: after building the ANN network and learning the network parameters through the Bayesian optimization algorithm, the fitted and trained substitute data M(t) can be input into the artificial neural network for prediction within a certain time, and the predicted internal tumor position is output

[0087] The evaluation index of the embodiment includes the performance of the neural network performance through the root mean square error (RMSE), the maximum error (ME). The specific formula is given as follows: ME represents the maximum error that occurs in the predicted respiratory sequence compared to the true sequence. RMSE is calculated by taking the square root of the average of the squares of all errors. RMSE represents the sample standard deviation of the difference between the predicted value and the true value. Wherein, and Y(t) represent the predicted and actual tumor positions, respectively.

[0088] Figure 7 is a schematic diagram of a visceral motion prediction device provided by an embodiment of the present application, as Figure 7 shown, the present application also provides a visceral motion prediction device, comprising: an acquisition module 71, a prediction module 72, a fitting module 73, which will be described in detail below.

[0089] The acquisition module 71 is configured to acquire magnetic induction data of the external skin of the internal organs at the current time, wherein the magnetic induction data comprises motion information of the external skin at different coordinates; the prediction module 72 is connected to the acquisition module 71 and configured to input the magnetic induction data into a trained prediction model, and output predicted magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time; the fitting module 73 is connected to the prediction module 72 and configured to input the predicted magnetic induction data into a trained fitting model, and output motion data of the internal organs from the fitting model.

[0090] The above-mentioned visceral motion prediction device acquires magnetic induction data of the external skin of the internal organs, predicts magnetic induction data at a target time through a prediction model, and then fits out internal organ motion data corresponding to the magnetic induction data at the target time using a fitting model. The purpose of determining the internal organ motion at the target time is achieved by collecting the magnetic induction data of the external skin of the internal organs at the current time, and the technical effect of obtaining the internal organ motion at the target time in advance is achieved, thereby solving the problem in the related art that the internal organ position changes due to the motion of the measured person such as breathing when the internal organ is imaged, resulting in that the imaged internal organ position is the internal organ position before the current time, and the internal organ motion cannot be obtained in time.

[0091] The visceral movement prediction device includes a processor and a memory. The acquisition module 71, prediction module 72, fitting module 73, etc., are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0092] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting the kernel parameters, the problem of users being unable to determine compatibility with non-capacitive screen original capacitive pens in related technologies can be solved.

[0093] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0094] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for predicting visceral movement.

[0095] This invention provides a processor for running a program, wherein the program executes a method for predicting visceral movement during runtime.

[0096] Figure 8 This is a schematic diagram of an electronic device provided according to an embodiment of this application, such as... Figure 8 As shown, this application embodiment provides an electronic device 80, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-mentioned method for predicting visceral movement:

[0097] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0098] This application also provides a computer program product that, when executed on a visceral motion prediction device, is adapted to execute a program that initializes any of the above-described method steps.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The computer program instructions can also be loaded onto a computer or other programmable internal organs of the prediction device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that the instructions that are executed on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks in the flowchart and / or block diagram.

[0101] The computer program instructions can also be loaded onto a computer or other programmable internal organs of the prediction device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that the instructions that are executed on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks in the flowchart and / or block diagram.

[0102] The computer program instructions can also be loaded onto a computer or other programmable internal organs of the prediction device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that the instructions that are executed on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks in the flowchart and / or block diagram.

[0103] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0104] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), or electrically erasable programmable read only memory (EEPROM), for the storage of software that is read during runtime. The memory is an example of computer readable media.

[0105] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0106] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but also other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0107] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0108] The above merely provides embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A method of predicting visceral movement, characterized by, The method comprises: acquiring magnetic induction data of the external skin of the internal organs at the current time, wherein the magnetic induction data comprises motion information of the external skin at different coordinates; inputting the magnetic induction data into a trained prediction model, and outputting predicted magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time; inputting the predicted magnetic induction data into a trained fitting model, and outputting motion data of the internal organs from the fitting model.

2. The method of claim 1, wherein, Before inputting the magnetic induction data into the trained prediction model and outputting the predicted magnetic induction data at the target time from the prediction model, the method further comprises: acquiring first historical data of the magnetic induction data, wherein the first historical data is magnetic induction data before the current time; extracting a plurality of sets of first training data from the first historical data, wherein each set of first training data comprises magnetic induction data at a preceding time as model input and magnetic induction data at a subsequent time as model output, and the time interval between the preceding time and the subsequent time is the same as the time interval between the current time and the target time; training the prediction model according to the plurality of sets of first training data until the prediction error of the prediction model reaches a first preset requirement.

3. The method of claim 2, wherein, After inputting the magnetic induction data into the trained prediction model and outputting the predicted magnetic induction data at the target time from the prediction model, the method further comprises: collecting target magnetic induction data at the target time; updating the first historical data according to the target magnetic induction data to update the corresponding training data; updating the prediction model according to the updated training data to obtain an updated prediction model.

4. The method of claim 2, wherein, Training the prediction model according to the plurality of sets of first training data until the prediction error of the prediction model reaches a first preset requirement comprises: creating an initial model according to a multi-layer perception network, wherein the multi-layer perception network comprises a plurality of network parameters, and the plurality of network parameters at least comprise at least one of the following: a prediction window input number, a hidden layer number, and the multi-layer perception network learns the network parameters through a Bayesian optimization algorithm; initializing the initial model, and performing supervised learning training on the initial model through the plurality of sets of first training data using a normal propagation algorithm; in a case where it is detected that the prediction error of the initial model meets the first preset requirement, taking the trained initial model as the prediction model, wherein the prediction error comprises a maximum error ME and a root mean square error RMSE.

5. The method of claim 1, wherein, Acquiring magnetic induction data of the external skin of the internal organs at the current time comprises: collecting raw magnetic induction data through an EM sensor arranged on the external skin; converting the raw magnetic induction data into a one-dimensional signal in a three-dimensional coordinate direction, and determining a first principal direction for marking the respiratory motion of the external skin according to the one-dimensional signal; performing data normalization, data smoothing, and abnormal value removal processing on the first principal direction to obtain the magnetic induction data.

6. The method of claim 1, wherein, Before inputting the predicted magnetic induction data into the trained fitting model and outputting the motion data of the internal organs from the fitting model, the method further comprises: obtain second historical data of the magnetic induction data, wherein the second historical data is magnetic induction data and ultrasound data synchronously collected in a historical time period; extract a plurality of sets of second training data from the second historical data, wherein each set of second training data comprises magnetic induction data and ultrasound data at the same time, the magnetic induction data as input and the ultrasound data as output; train the fitting model according to the plurality of sets of second training data until a fitting error of the fitting model reaches a second preset requirement.

7. The method of claim 6, wherein, obtaining second historical data of the magnetic induction data comprises: obtaining ultrasound data of the internal organs collected historically and magnetic induction data of the external skin collected historically; wherein the ultrasound data of the internal organs collected historically is collected at a time when collecting ultrasound images of the internal organs; registering two adjacent ultrasound images to obtain offset information of deformation field pixels; calculating offset motion information according to the offset information; performing data normalization, data smoothing and abnormal value removal processing on the offset motion information to obtain the ultrasound data.

8. The method of claim 6, wherein, training the fitting model according to the plurality of sets of second training data until a fitting error of the fitting model reaches a second preset requirement comprises: creating a motion model by a multivariate linear regression method, wherein the motion model is as follows: where (t) is the ultrasound data representing the visceral motion, M(t) i represents the i-th magnetic induction data parameter, n represents the number of parameters of the magnetic induction data, M(t)0is a virtual unit intercept for representing a bias of the motion model, β i is a coefficient of the respective magnetic induction data parameter; determining the fitting error according to the plurality of sets of second training data, wherein the fitting error comprises a root mean square error (RMSE); in a case where the fitting error reaches the second preset requirement, taking the trained motion model as the fitting model.

9. A device for predicting visceral movement, characterized in that, comprise: a obtaining module configured to obtain magnetic induction data of internal organs and external skin at a current time, wherein the magnetic induction data comprises motion information of the external skin at different coordinates; a predicting module configured to input the magnetic induction data into a trained prediction model and output predicted magnetic induction data at a target time from the prediction model, wherein the target time is a time after the current time; a fitting module configured to input the predicted magnetic induction data into a trained fitting model and output motion data of the internal organs from the fitting model.

10. An electronic device, comprising: comprise one or more processors and a memory, the memory being configured to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method for predicting motion of internal organs according to any one of claims 1 to 8.

Images