一种面向胸腹部放疗的实时呼吸运动预测与补偿方法

Abstract

本发明公开一种面向胸腹部放疗的实时呼吸运动预测与补偿方法，包括呼吸信号采集模块、呼吸运动预测模块、体表标记点位置监测模块、体表标记点位移预测模块、体表标记点坐标转换模块、肿瘤靶区中心点预测模块、肿瘤靶区可视化模块；本发明公开了一种基于因果卷积网络的呼吸运动预测系统，本发明通过实时采集患者的呼吸波形信号和体表标记点运动轨迹，提取周期性运动模式，预测未来0.5‑2秒的肿瘤靶区三维位移量，实现未来肿瘤靶区位置的精准预测。

Description

Technical Field

[0001] This invention relates to the field of medical technology, specifically a method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy. Background Technology

[0002] Currently, the main treatment for lung cancer is stereotactic radiotherapy. However, respiratory motion significantly affects the accuracy of radiotherapy. To reduce the impact of respiratory motion, the most effective method is real-time respiratory motion tracking technology. This technology establishes a correlation model between the tumor inside the body and surface markers, and uses predictive algorithms to obtain future tumor movement information based on the movement of the markers. This allows for real-time adjustment of the radiation beam, ensuring relative stillness between the radiation and the tumor, thus achieving real-time tracking. The predictive algorithm is used to compensate for the time delay of the stereotactic radiotherapy system, predicting in advance the location the tumor will reach at a future moment.

[0003] However, existing algorithms do not take into account the impact of continuous respiratory fluctuations on changes in the location of tumors and surface markers in vivo. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time respiratory motion prediction and compensation method for thoracic and abdominal radiotherapy. This method achieves real-time respiratory motion prediction through a real-time respiratory motion prediction and compensation system.

[0005] The real-time respiratory motion prediction and compensation system includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate transformation module, a tumor target center point prediction module, and a tumor target visualization module.

[0006] The steps to achieve real-time respiratory motion prediction include:

[0007] 1) The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed. t And transmit it to the respiratory motion prediction module;

[0008] 2) The respiratory motion prediction module stores a respiratory motion prediction model based on an LSTM network;

[0009] The respiratory motion prediction module is based on the patient's real-time respiratory signal A. t Construct a respiratory signal sequence with respiratory signals from the past T0 time period. Then, the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the respiratory signal sequence for the future T1 time period. and respiratory state; the respiratory state is exhalation or inhalation;

[0010] The respiratory motion prediction module will generate the respiratory signal sequence for the future T1 time period. The respiratory status is transmitted to the body surface marker displacement prediction module, and the respiratory status is transmitted to the tumor target center point prediction module.

[0011] 3) The body surface marker position monitoring module acquires the real-time three-dimensional world coordinates (X, Y, F, Z) of the body surface markers. t ,Y t Z t The data is then transmitted to the body surface marker displacement prediction module.

[0012] The body surface marker displacement prediction module stores a body surface marker prediction model.

[0013] The body surface marker displacement prediction module will use the patient's respiratory signal sequence at the future T1 time period. The respiratory state is input into the displacement prediction model of the body surface markers to obtain the three-dimensional displacement change sequence d={(d x1 ,d y1 ,d z1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) The three-dimensional displacement change of the surface marker point at time T1 is determined based on the three-dimensional displacement change sequence d.

[0014] 4) The surface marker displacement prediction module combines the real-time three-dimensional world coordinates and three-dimensional displacement change (d) of the surface markers. x ,d y ,d z Using the three-dimensional displacement change sequence d, the three-dimensional world coordinates of the surface marker points at time T1 in the future are determined. And the three-dimensional world coordinate sequence of the body surface markers in the future time period T1, thereby constructing the three-dimensional world coordinate prediction sequence F of the body surface markers, that is:

[0015]

[0016] 5) The body surface marker point coordinate transformation module stores the camera intrinsic parameter matrix and extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0017] The body surface marker coordinate transformation module converts the three-dimensional world coordinate prediction sequence F of the body surface markers into the image coordinate sequence F' of the body surface markers in the future T1 time period based on the camera intrinsic and extrinsic parameter matrices, and transmits it to the tumor target center point prediction module.

[0018] The image coordinate sequence F' of the body surface marker points in the future time period T1 is shown below:

[0019]

[0020] In the formula, u and v are image coordinates; s is the depth information of the body surface marker point;

[0021] 6) The tumor target center point prediction module stores tumor target center point prediction models under different respiratory states;

[0022] The tumor target center point prediction module calls the corresponding tumor target center point prediction model based on the patient's respiratory status, and uses the called tumor target center point prediction model to process the image coordinate sequence F' of the body surface marker points at the future time T1, to obtain the predicted image coordinates (Q) of the tumor target center point at the future time T1. x Q y Q z );Q z For depth information of the center point of the tumor target area;

[0023] 7) The tumor target visualization module will predict the image coordinates (Q) of the center point of the tumor target at time T1 in the future. x Q y Q z Visualization;

[0024] 8) After T1 time, obtain the actual respiratory signal sequence for the future T1 time period. The actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}、The three-dimensional world coordinates of the body surface marker at time T1 in the future Future time T1 body surface marker image coordinates Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z );

[0025] Then, using actual respiratory signal sequences The respiratory motion prediction model based on LSTM network is updated online;

[0026] Using actual respiratory signal sequences and the actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) The displacement prediction model for surface markers is updated online.

[0027] Using the three-dimensional world coordinates of the body surface markers at time T1 in the future Future time T1 body surface marker image coordinates Update the camera extrinsic matrix online;

[0028] Using the image coordinates of the body surface markers at time T1 in the future Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z The tumor target center point prediction model is updated online.

[0029] The steps for constructing the body surface marker prediction model are as follows:

[0030] a1) Monitor the patient's respiratory signal within time Tmax, and divide the respiratory signal into multiple respiratory signal sequences of duration T1 according to a preset step size Δt, and construct a respiratory fluctuation matrix B, i.e.:

[0031]

[0032] In the formula, For respiratory fluctuations The start and end times; For respiratory fluctuations Corresponding respiratory state; respiratory signal sequence

[0033] At the start and end of each respiratory fluctuation, the real-time three-dimensional world coordinates (X, Y, Z) of the surface markers are obtained using the surface marker position monitoring module, and the three-dimensional displacement change sequence D = {(D x1 D y1 D z1 ),(D x2 D y2 D z2 ),...,(D x(n-1) D y(n-1) D z(n-1) )};

[0034] a2) Repeat step a1) to obtain multiple sets of respiratory fluctuation-three-dimensional displacement change sample pairs, thereby constructing a training sample set C for respiratory fluctuation-three-dimensional displacement change, i.e.:

[0035]

[0036] In the formula, This is a respiratory signal sequence; In a breathing state; D EG ={(D EGx1 D EGy1 D EGz1 ),(D EGx2 D EGy2 D EGz2 ),...,(D EGx(n-1) D EGy(n-1) D EGz(n-1) )} is a respiratory signal sequence The corresponding sequence of three-dimensional displacement changes at body surface markers;

[0037] a3) Construct a hybrid network model based on LSTM and TCN, including an input layer, a TCN module, a bidirectional LSTM module, a multi-head attention mechanism module, and an output layer;

[0038] The input layer takes a respiratory signal sequence as input.

[0039] The TCN module includes at least four stacked dilated causal convolutional layers for extracting local morphological and periodic features of respiratory fluctuations.

[0040] The bidirectional LSTM module includes two stacked LSTM layers, used to construct the dependency between respiratory signals and the three-dimensional displacement changes of surface markers.

[0041] The multi-head attention mechanism module is used to enhance the influence weight of respiratory peaks and troughs on the three-dimensional displacement changes of body surface markers.

[0042] The multi-head attention mechanism MultiHead(Q,K,V) is as follows:

[0043] MultiHead(Q,K,V)=Concat(head1,...,head h (5)

[0044] head i =Attention(QW i Q ,KW i K VW iV (6)

[0045]

[0046] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; W is the scaling factor. i Q W i K W i V The projection matrix is ​​represented by `Q`; `Attention(Q,K,V)` represents the attention weights; the superscript `T` indicates transpose; `head` represents the projected matrix. i This represents the i-th attention computation unit;

[0047] The output layer is used to output the three-dimensional displacement change sequence of the body surface marker points;

[0048] a4) The hybrid network model based on LSTM and TCN was trained and validated using the training sample set C of respiratory fluctuation-three-dimensional displacement change to obtain the body surface marker prediction model.

[0049] The body surface marker coordinate transformation module will... 3D world coordinates of the body surface marker at time Transform into the future Image coordinates of surface markers at time The steps include:

[0050] b1) Convert the three-dimensional world coordinates Convert to camera coordinates, i.e.:

[0051]

[0052] In the formula, R1 and R2 are the camera coordinates; R1 and R2 are the rotation matrix (representing the camera pose) and translation vector (representing the camera position).

[0053] The rotation matrix R1 is shown below:

[0054]

[0055] b2) Set camera coordinates Convert to image coordinates Right now:

[0056]

[0057] In the formula, depth information

[0058] The intrinsic parameter matrix K is shown below:

[0059]

[0060] In the formula, f x f y c is the focal length; x c y The coordinates of the image center point;

[0061] The steps for constructing a tumor target center point prediction model are as follows:

[0062] c1) Under the same respiratory state, acquire images of surface markers and in vivo tumor target areas at time T1;

[0063] c2) Extract the image coordinate sequence of the surface markers at time T1 and the image coordinates of the center point of the tumor target area in the body at time T1, and construct the mapping relationship;

[0064] c3) Repeat steps c1)-c2) to obtain the tumor target center point prediction training sample set M, i.e.:

[0065]

[0066] In the formula, The coordinates of the marked points on the body surface; For depth information of surface markers; (M 2mx M 2my ) represents the coordinates of the center point of the tumor target area in vivo; M 2mz For depth information of the center point of the tumor target area in vivo; M 1m A sequence of image coordinates for marker points on the body surface;

[0067] c3) Construct a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, gated activation layers, skip connection layers, and an output layer;

[0068] Among them, the dilated causal convolutional layer is used to extract the image coordinate sequence features of the body surface markers during time T1;

[0069] At time step t, the output of the dilated causal convolutional layer As shown below:

[0070]

[0071] In the formula, ω l d is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolutional layers; x t-d·l To expand the input of the causal convolutional layer;

[0072] The output of the residual block is shown below:

[0073] E'=w2σ(w1x')+w3x'(14)

[0074] In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are the residual block parameters; σ is the ReLU activation function; w3 is the linear transformation function, used to ensure that the dimensions of two adjacent residual blocks are consistent;

[0075] The skip connection layer transmits information from different layers to the output layer;

[0076] The output layer is used to output the image coordinates of the center point of the tumor target area in vivo at time T1.

[0077] c4) The bidirectional causal convolution model is trained using the tumor target center point prediction training sample set to obtain the tumor target center point prediction model.

[0078] Furthermore, respiratory signals are acquired using a laser detector or a binocular camera.

[0079] Furthermore, the value range of T1 is [0.5s, 2s].

[0080] Furthermore, the respiratory motion prediction model includes multiple LSTM layers;

[0081] During the forward propagation of the LSTM network, the output h of each LSTM unit t As shown below:

[0082] f t =σ(W f ·[h t-1 ,x t ]+b f (15)

[0083] i t =σ(W i ·[h t-1 ,x t ]+b i (16)

[0084]

[0085] o t =σ(W o ·[h t-1 ,x t ]+b o (19)

[0086] h t =o t *tanh(C t (20)

[0087] In the formula, x t This is the current input, h t-1 It is the output of the previous time step, f t i t o t These are the outputs of the forget gate, input gate, and output gate, respectively. It is a candidate cell state, C t It is a cellular state, W f W i W C W o and b f b i b C b o These are the weights and biases that the network needs to learn; σ is the activation function; C t-1 It represents the cell state at the previous moment.

[0088] Furthermore, the respiratory motion prediction model based on the LSTM network is constructed through the following steps:

[0089] d1) Collect T data from patients on the treatment bed max The respiratory signals are collected over a period of time and then divided into multiple input-output sample pairs to construct the training sample set for the LSTM network. Each input sample has a time length of T0+1, and the output sample is the respiratory signal at time T after the input sample.

[0090] d2) Train the LSTM network using the LSTM network training sample set to obtain the respiratory motion prediction model based on the LSTM network;

[0091] During training, the weights and bias parameters in the LSTM network are updated using gradient descent. The updated parameters are shown below:

[0092]

[0093] In the formula, W′ and b′ are the updated weights and biases; α is the learning rate; L is the loss function; and W and b are the weights and biases before the update.

[0094] Furthermore, when training the hybrid network model based on LSTM and TCN, the Adam optimizer is used to optimize the parameters of the hybrid network model after each training iteration.

[0095] Furthermore, the body surface marker location monitoring module is an electromagnetic sensor attached to the location of the marker on the patient's body surface.

[0096] Furthermore, during training, the loss function used for the LSTM network and the hybrid network model based on LSTM and TCN is the mean absolute error loss.

[0097] Furthermore, the camera extrinsic matrix is ​​obtained through calibration.

[0098] The technical effects of this invention are beyond doubt. This invention discloses a respiratory motion prediction system based on causal convolutional networks. This invention extracts periodic motion patterns by real-time acquisition of the patient's respiratory waveform signals and the motion trajectory of surface markers, and predicts the three-dimensional displacement of the tumor target area in the next 0.5-2 seconds, thereby achieving accurate prediction of the future location of the tumor target area. Attached Figure Description

[0099] Figure 1 This is a schematic diagram of the system modules;

[0100] Figure 2 This is a flowchart of the method. Detailed Implementation

[0101] The present invention will be further described below with reference to embodiments, but it should not be construed that the scope of the present invention is limited to the following embodiments. Various substitutions and modifications made based on ordinary technical knowledge and common practices in the art without departing from the above-described technical concept of the present invention should be included within the scope of protection of the present invention.

[0102] Example 1:

[0103] See Figures 1-2 This method achieves real-time respiratory motion prediction through a real-time respiratory motion prediction and compensation system;

[0104] The real-time respiratory motion prediction and compensation system includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate transformation module, a tumor target center point prediction module, and a tumor target visualization module.

[0105] The steps to achieve real-time respiratory motion prediction include:

[0106] 1) The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed. t And transmit it to the respiratory motion prediction module;

[0107] 2) The respiratory motion prediction module stores a respiratory motion prediction model based on an LSTM network;

[0108] The respiratory motion prediction module is based on the patient's real-time respiratory signal A. t Construct a respiratory signal sequence with respiratory signals from the past T0 time period. Then, the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the respiratory signal sequence for the future T1 time period. and respiratory state; the respiratory state is exhalation or inhalation;

[0109] The respiratory motion prediction module will generate the respiratory signal sequence for the future T1 time period. The respiratory status is transmitted to the body surface marker displacement prediction module, and the respiratory status is transmitted to the tumor target center point prediction module.

[0110] 3) The body surface marker position monitoring module acquires the real-time three-dimensional world coordinates (X, Y, F, Z) of the body surface markers. t ,Y t Z t The data is then transmitted to the body surface marker displacement prediction module.

[0111] The body surface marker displacement prediction module stores a body surface marker prediction model.

[0112] The body surface marker displacement prediction module will use the patient's respiratory signal sequence at the future T1 time period. The respiratory state is input into the displacement prediction model of the body surface markers to obtain the three-dimensional displacement change sequence d={(d x1 ,d y1 ,d z1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) The three-dimensional displacement change of the surface marker point at time T1 is determined based on the three-dimensional displacement change sequence d.

[0113] 4) The surface marker displacement prediction module combines the real-time three-dimensional world coordinates and three-dimensional displacement change (d) of the surface markers. x ,d y ,d z Using the three-dimensional displacement change sequence d, the three-dimensional world coordinates of the surface marker points at time T1 in the future are determined. And the three-dimensional world coordinate sequence of the body surface markers in the future time period T1, thereby constructing the three-dimensional world coordinate prediction sequence F of the body surface markers, that is:

[0114]

[0115] 5) The body surface marker point coordinate transformation module stores the camera intrinsic parameter matrix and extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0116] The body surface marker coordinate transformation module converts the three-dimensional world coordinate prediction sequence F of the body surface markers into the image coordinate sequence F' of the body surface markers in the future T1 time period based on the camera intrinsic and extrinsic parameter matrices, and transmits it to the tumor target center point prediction module.

[0117] The image coordinate sequence F' of the body surface marker points in the future time period T1 is shown below:

[0118]

[0119] In the formula, u and v are image coordinates; s is the depth information of the body surface marker point;

[0120] 6) The tumor target center point prediction module stores tumor target center point prediction models under different respiratory states;

[0121] The tumor target center point prediction module calls the corresponding tumor target center point prediction model based on the patient's respiratory status, and uses the called tumor target center point prediction model to process the image coordinate sequence F' of the body surface marker points at the future time T1, to obtain the predicted image coordinates (Q) of the tumor target center point at the future time T1. x Q y Q z );Q z For depth information of the center point of the tumor target area;

[0122] 7) The tumor target visualization module will predict the image coordinates (Q) of the center point of the tumor target at time T1 in the future. x Q y Q z Visualization;

[0123] 8) After T1 time, obtain the actual respiratory signal sequence for the future T1 time period. The actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}、The three-dimensional world coordinates of the body surface marker at time T1 in the future Future time T1 body surface marker image coordinates Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z );

[0124] Then, using actual respiratory signal sequences The respiratory motion prediction model based on LSTM network is updated online;

[0125] Using actual respiratory signal sequences and the actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) The displacement prediction model for surface markers is updated online.

[0126] Using the three-dimensional world coordinates of the body surface markers at time T1 in the future Future time T1 body surface marker image coordinates Update the camera extrinsic matrix online;

[0127] Using the image coordinates of the body surface markers at time T1 in the future Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z The tumor target center point prediction model is updated online.

[0128] The steps for constructing the body surface marker prediction model are as follows:

[0129] a1) Monitor the patient's respiratory signal within time Tmax, and divide the respiratory signal into multiple respiratory signal sequences of duration T1 according to a preset step size Δt, and construct a respiratory fluctuation matrix B, i.e.:

[0130]

[0131] In the formula, For respiratory fluctuations The start and end times; For respiratory fluctuations Corresponding respiratory state; respiratory signal sequence

[0132] At the start and end of each respiratory fluctuation, the real-time three-dimensional world coordinates (X, Y, Z) of the surface markers are obtained using the surface marker position monitoring module, and the three-dimensional displacement change sequence D = {(D x1 D y1 D z1 ),(Dx2 D y2 D z2 ),...,(D x(n-1) D y(n-1) D z(n-1) )};

[0133] a2) Repeat step a1) to obtain multiple sets of respiratory fluctuation-three-dimensional displacement change sample pairs, thereby constructing a training sample set C for respiratory fluctuation-three-dimensional displacement change, i.e.:

[0134]

[0135] In the formula, This is a respiratory signal sequence; In a breathing state; D EG ={(D EGx1 D EGy1 D EGz1 ),(D EGx2 D EGy2 D EGz2 ),...,(D EGx(n-1) D EGy(n-1) D EGz(n-1) )} is a respiratory signal sequence The corresponding sequence of three-dimensional displacement changes at body surface markers;

[0136] a3) Construct a hybrid network model based on LSTM and TCN, including an input layer, a TCN module, a bidirectional LSTM module, a multi-head attention mechanism module, and an output layer;

[0137] The input layer takes a respiratory signal sequence as input.

[0138] The TCN module includes at least four stacked dilated causal convolutional layers for extracting local morphological and periodic features of respiratory fluctuations.

[0139] The bidirectional LSTM module includes two stacked LSTM layers, used to construct the dependency between respiratory signals and the three-dimensional displacement changes of surface markers.

[0140] The multi-head attention mechanism module is used to enhance the influence weight of respiratory peaks and troughs on the three-dimensional displacement changes of body surface markers.

[0141] The multi-head attention mechanism MultiHead(Q,K,V) is as follows:

[0142] MultiHead(Q,K,V)=Concat(head1,...,head h (5)

[0143] headi =Attention(QW i Q ,KW i K VW i V (6)

[0144]

[0145] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; W is the scaling factor. i Q W i K W i V The projection matrix is ​​represented by `Q`; `Attention(Q,K,V)` represents the attention weights; the superscript `T` indicates transpose; `head` represents the projected matrix. i This represents the i-th attention computation unit;

[0146] The output layer is used to output the three-dimensional displacement change sequence of the body surface marker points;

[0147] a4) The hybrid network model based on LSTM and TCN was trained and validated using the training sample set C of respiratory fluctuation-three-dimensional displacement change to obtain the body surface marker prediction model.

[0148] The body surface marker coordinate transformation module will... 3D world coordinates of the body surface marker at time Transform into the future Image coordinates of surface markers at time The steps include:

[0149] b1) Convert the three-dimensional world coordinates Convert to camera coordinates, i.e.:

[0150]

[0151] In the formula, R1 and R2 are the camera coordinates; R1 and R2 are the rotation matrix (representing the camera pose) and translation vector (representing the camera position).

[0152] The rotation matrix R1 is shown below:

[0153]

[0154] b2) Set camera coordinates Convert to image coordinates Right now:

[0155]

[0156] In the formula, depth information

[0157] The intrinsic parameter matrix K is shown below:

[0158]

[0159] In the formula, f x f y c is the focal length; x c y The coordinates of the image center point;

[0160] The steps for constructing a tumor target center point prediction model are as follows:

[0161] c1) Under the same respiratory state, acquire images of surface markers and in vivo tumor target areas at time T1;

[0162] c2) Extract the image coordinate sequence of the surface markers at time T1 and the image coordinates of the center point of the tumor target area in the body at time T1, and construct the mapping relationship;

[0163] c3) Repeat steps c1)-c2) to obtain the tumor target center point prediction training sample set M, i.e.:

[0164]

[0165] In the formula, The coordinates of the marked points on the body surface; For depth information of surface markers; (M 2mx M 2my ) represents the coordinates of the center point of the tumor target area in vivo; M 2mz For depth information of the center point of the tumor target area in vivo; M 1m A sequence of image coordinates for marker points on the body surface;

[0166] c3) Construct a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, gated activation layers, skip connection layers, and an output layer;

[0167] Among them, the dilated causal convolutional layer is used to extract the image coordinate sequence features of the body surface markers during time T1;

[0168] At time step t, the output of the dilated causal convolutional layer As shown below:

[0169]

[0170] In the formula, ω ld is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolutional layers; x t-d·l To expand the input of the causal convolutional layer;

[0171] The output of the residual block is shown below:

[0172] E'=w2σ(w1x')+w3x'(14)

[0173] In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are the residual block parameters; σ is the ReLU activation function; w3 is the linear transformation function, used to ensure that the dimensions of two adjacent residual blocks are consistent;

[0174] The skip connection layer transmits information from different layers to the output layer;

[0175] The output layer is used to output the image coordinates of the center point of the tumor target area in vivo at time T1.

[0176] c4) The bidirectional causal convolution model is trained using the tumor target center point prediction training sample set to obtain the tumor target center point prediction model.

[0177] Respiratory signals are collected using a laser detector or a binocular camera.

[0178] The value range of T1 is [0.5s, 2s].

[0179] The respiratory motion prediction model includes multiple LSTM layers;

[0180] During the forward propagation of the LSTM network, the output h of each LSTM unit t As shown below:

[0181] f t =σ(W f ·[h t-1 ,x t ]+b f (15)

[0182] i t =σ(W i ·[h t-1 ,x t ]+b i (16)

[0183]

[0184] o t =σ(W o ·[h t-1 ,x t ]+b o (19)

[0185] h t =o t *tanh(C t (20)

[0186] In the formula, x t This is the current input, h t-1 It is the output of the previous time step, f t i t o t These are the outputs of the forget gate, input gate, and output gate, respectively. It is a candidate cell state, C t It is a cellular state, W f W i W C W o and b f b i b C b o These are the weights and biases that the network needs to learn; σ is the activation function; C t-1 It represents the cell state at the previous moment.

[0187] The respiratory motion prediction model based on LSTM network is constructed through the following steps:

[0188] d1) Collect T data from patients on the treatment bed max The respiratory signals are collected over a period of time and then divided into multiple input-output sample pairs to construct the training sample set for the LSTM network. Each input sample has a time length of T0+1, and the output sample is the respiratory signal at time T after the input sample.

[0189] d2) Train the LSTM network using the LSTM network training sample set to obtain the respiratory motion prediction model based on the LSTM network;

[0190] During training, the weights and bias parameters in the LSTM network are updated using gradient descent. The updated parameters are shown below:

[0191]

[0192] In the formula, W′ and b′ are the updated weights and biases; α is the learning rate; L is the loss function; and W and b are the weights and biases before the update.

[0193] During the operation of the real-time respiratory motion prediction and compensation system, the actual respiratory signal sequence for the future T1 time period is acquired. The actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',dz1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}、The three-dimensional world coordinates of the body surface marker at time T1 in the future Future time T1 body surface marker image coordinates Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z );

[0194] Then, using actual respiratory signal sequences The respiratory motion prediction model based on LSTM network is updated online;

[0195] Using actual respiratory signal sequences and the actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) The displacement prediction model for surface markers is updated online.

[0196] Using the three-dimensional world coordinates of the body surface markers at time T1 in the future Future time T1 body surface marker image coordinates Update the camera extrinsic matrix online;

[0197] Using the image coordinates of the body surface markers at time T1 in the future Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z The tumor target center point prediction model is updated online.

[0198] When training a hybrid network model based on LSTM and TCN, the Adam optimizer is used to optimize the parameters of the hybrid network model after each training iteration.

[0199] The body surface marker location monitoring module is an electromagnetic sensor that is attached to the location of the marker on the patient's body surface.

[0200] During training, the loss function used for LSTM networks and hybrid network models based on LSTM and TCN is the mean absolute error loss.

[0201] The camera extrinsic matrix is ​​obtained through calibration.

[0202] Example 2:

[0203] A real-time respiratory motion prediction and compensation system for thoracic and abdominal radiotherapy includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate transformation module, a tumor target center point prediction module, and a tumor target visualization module.

[0204] The respiratory signal acquisition module collects real-time respiratory signals A from the patient on the treatment bed. t And transmit it to the respiratory motion prediction module;

[0205] The respiratory motion prediction module stores a respiratory motion prediction model based on an LSTM network.

[0206] The respiratory motion prediction module is based on the patient's real-time respiratory signal A. t Construct a respiratory signal sequence with respiratory signals from the past T0 time period. Then, the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the respiratory signal sequence for the future T1 time period. and respiratory state; the respiratory state is exhalation or inhalation;

[0207] The respiratory motion prediction module will generate the respiratory signal sequence for the future T1 time period. The respiratory status is transmitted to the body surface marker displacement prediction module, and the respiratory status is transmitted to the tumor target center point prediction module.

[0208] The body surface marker location monitoring module acquires the real-time three-dimensional world coordinates (X) of the body surface markers. t ,Y t Z t The data is then transmitted to the body surface marker displacement prediction module.

[0209] The body surface marker displacement prediction module stores a body surface marker prediction model.

[0210] The body surface marker displacement prediction module will use the patient's respiratory signal sequence at the future T1 time period. The respiratory state is input into the displacement prediction model of the body surface markers to obtain the three-dimensional displacement change sequence d={(d x1 ,d y1 ,d z1 ),(d x2 ,dy2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) The three-dimensional displacement change of the surface marker point at time T1 is determined based on the three-dimensional displacement change sequence d.

[0211] The body surface marker displacement prediction module combines the real-time three-dimensional world coordinates and three-dimensional displacement change (d) of the body surface markers. x ,d y ,d z Using the three-dimensional displacement change sequence d, the three-dimensional world coordinates of the surface marker points at time T1 in the future are determined. And the three-dimensional world coordinate sequence of the body surface markers in the future time period T1, thereby constructing the three-dimensional world coordinate prediction sequence F of the body surface markers, that is:

[0212]

[0213] The body surface marker point coordinate transformation module stores the camera intrinsic parameter matrix and extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector.

[0214] The body surface marker coordinate transformation module converts the three-dimensional world coordinate prediction sequence F of the body surface markers into the image coordinate sequence F' of the body surface markers in the future T1 time period based on the camera intrinsic and extrinsic parameter matrices, and transmits it to the tumor target center point prediction module.

[0215] The image coordinate sequence F' of the body surface marker points in the future time period T1 is shown below:

[0216]

[0217] In the formula, u and v are image coordinates; s is the depth information of the body surface marker point;

[0218] The tumor target center point prediction module stores tumor target center point prediction models under different respiratory states.

[0219] The tumor target center point prediction module calls the corresponding tumor target center point prediction model based on the patient's respiratory status, and uses the called tumor target center point prediction model to process the image coordinate sequence F' of the body surface marker points at the future time T1, to obtain the predicted image coordinates (Q) of the tumor target center point at the future time T1. x Q y Q z );Q z For depth information of the center point of the tumor target area;

[0220] The tumor target visualization module will predict the image coordinates (Q) of the center point of the tumor target at time T1 in the future. x Q y Q z Visualization;

[0221] The steps for constructing the body surface marker prediction model are as follows:

[0222] a1) Monitor the patient's respiratory signal within time Tmax, and divide the respiratory signal into multiple respiratory signal sequences of duration T1 according to a preset step size Δt, and construct a respiratory fluctuation matrix B, i.e.:

[0223]

[0224] In the formula, For respiratory fluctuations The start and end times; For respiratory fluctuations Corresponding respiratory state; respiratory signal sequence

[0225] At the start and end of each respiratory fluctuation, the real-time three-dimensional world coordinates (X, Y, Z) of the surface markers are obtained using the surface marker position monitoring module, and the three-dimensional displacement change sequence D = {(D x1 D y1 D z1 ),(D x2 D y2 D z2 ),...,(D x(n-1) D y(n-1) D z(n-1) )};

[0226] a2) Repeat step a1) to obtain multiple sets of respiratory fluctuation-three-dimensional displacement change sample pairs, thereby constructing a training sample set C for respiratory fluctuation-three-dimensional displacement change, i.e.:

[0227]

[0228] In the formula, This is a respiratory signal sequence; In a breathing state; D EG ={(D EGx1 D EGy1 D EGz1 ),(D EGx2 D EGy2 D EGz2 ),...,(D EGx(n-1) D EGy(n-1) D EGz(n-1))} is a respiratory signal sequence The corresponding sequence of three-dimensional displacement changes at body surface markers;

[0229] a3) Construct a hybrid network model based on LSTM and TCN, including an input layer, a TCN module, a bidirectional LSTM module, a multi-head attention mechanism module, and an output layer;

[0230] The input layer takes a respiratory signal sequence as input.

[0231] The TCN module includes at least four stacked dilated causal convolutional layers for extracting local morphological and periodic features of respiratory fluctuations.

[0232] The bidirectional LSTM module includes two stacked LSTM layers, used to construct the dependency between respiratory signals and the three-dimensional displacement changes of surface markers.

[0233] The multi-head attention mechanism module is used to enhance the influence weight of respiratory peaks and troughs on the three-dimensional displacement changes of body surface markers.

[0234] The multi-head attention mechanism MultiHead(Q,K,V) is as follows:

[0235] MultiHead(Q,K,V)=Concat(head1,...,head h (5)

[0236] head i =Attention(QW i Q ,KW i K VW i V (6)

[0237]

[0238] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; W is the scaling factor. i Q W i K W i V The projection matrix is ​​represented by `Q`; `Attention(Q,K,V)` represents the attention weights; the superscript `T` indicates transpose; `head` represents the projected matrix. i This represents the i-th attention computation unit;

[0239] The output layer is used to output the three-dimensional displacement change sequence of the body surface marker points;

[0240] a4) The hybrid network model based on LSTM and TCN was trained and validated using the training sample set C of respiratory fluctuation-three-dimensional displacement change to obtain the body surface marker prediction model.

[0241] The body surface marker coordinate transformation module will... 3D world coordinates of the body surface marker at time Transform into the future Image coordinates of surface markers at time The steps include:

[0242] b1) Convert the three-dimensional world coordinates Convert to camera coordinates, i.e.:

[0243]

[0244] In the formula, R1 and R2 are the camera coordinates; R1 and R2 are the rotation matrix (representing the camera pose) and translation vector (representing the camera position).

[0245] The rotation matrix R1 is shown below:

[0246]

[0247] b2) Set camera coordinates Convert to image coordinates Right now:

[0248]

[0249] In the formula, depth information

[0250] The intrinsic parameter matrix K is shown below:

[0251]

[0252] In the formula, f x f y c is the focal length; x c y The coordinates of the image center point;

[0253] The steps for constructing a tumor target center point prediction model are as follows:

[0254] c1) Under the same respiratory state, acquire images of surface markers and in vivo tumor target areas at time T1;

[0255] c2) Extract the image coordinate sequence of the surface markers at time T1 and the image coordinates of the center point of the tumor target area in the body at time T1, and construct the mapping relationship;

[0256] c3) Repeat steps c1)-c2) to obtain the tumor target center point prediction training sample set M, i.e.:

[0257]

[0258] In the formula, The coordinates of the marked points on the body surface; For depth information of surface markers; (M 2mx M 2my ) represents the coordinates of the center point of the tumor target area in vivo; M 2mz For depth information of the center point of the tumor target area in vivo; M 1m A sequence of image coordinates for marker points on the body surface;

[0259] c3) Construct a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, gated activation layers, skip connection layers, and an output layer;

[0260] Among them, the dilated causal convolutional layer is used to extract the image coordinate sequence features of the body surface markers during time T1;

[0261] At time step t, the output of the dilated causal convolutional layer As shown below:

[0262]

[0263] In the formula, ω l d is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolutional layers; x t-d·l To expand the input of the causal convolutional layer;

[0264] The output of the residual block is shown below:

[0265] E'=w2σ(w1x')+w3x'(14)

[0266] In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are the residual block parameters; σ is the ReLU activation function; w3 is the linear transformation function, used to ensure that the dimensions of two adjacent residual blocks are consistent;

[0267] The skip connection layer transmits information from different layers to the output layer;

[0268] The output layer is used to output the image coordinates of the center point of the tumor target area in vivo at time T1.

[0269] c4) The bidirectional causal convolution model is trained using the tumor target center point prediction training sample set to obtain the tumor target center point prediction model.

[0270] Example 3:

[0271] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-2, further wherein the respiratory signal is acquired by a laser detector or a binocular camera.

[0272] Example 4:

[0273] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-3, further wherein the value range of T1 is [0.5s, 2s].

[0274] Example 5:

[0275] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-4, further wherein the respiratory motion prediction model includes multiple LSTM layers;

[0276] During the forward propagation of the LSTM network, the output h of each LSTM unit t As shown below:

[0277] f t =σ(W f ·[h t-1 ,x t ]+b f (15)

[0278] i t =σ(W i ·[h t-1 ,x t ]+b i (16)

[0279]

[0280] o t =σ(W o ·[h t-1 ,x t ]+b o (19)

[0281] h t =o t *tanh(C t (20)

[0282] In the formula, x t This is the current input, h t-1 It is the output of the previous time step, f t i t o t These are the outputs of the forget gate, input gate, and output gate, respectively. It is a candidate cell state, C t It is a cellular state, W f W i W C W o and b f b i b C b o These are the weights and biases that the network needs to learn; σ is the activation function; C t-1 It represents the cell state at the previous moment.

[0283] Example 6:

[0284] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-5, further wherein the respiratory motion prediction model based on LSTM network is constructed through the following steps:

[0285] d1) Collect T data from patients on the treatment bed max The respiratory signals are collected over a period of time and then divided into multiple input-output sample pairs to construct the training sample set for the LSTM network. Each input sample has a time length of T0+1, and the output sample is the respiratory signal at time T after the input sample.

[0286] d2) Train the LSTM network using the LSTM network training sample set to obtain the respiratory motion prediction model based on the LSTM network;

[0287] During training, the weights and bias parameters in the LSTM network are updated using gradient descent. The updated parameters are shown below:

[0288]

[0289] In the formula, W′ and b′ are the updated weights and biases; α is the learning rate; L is the loss function; and W and b are the weights and biases before the update.

[0290] Example 7:

[0291] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-6, further comprising acquiring the actual respiratory signal sequence for the future T1 time period during the operation of the real-time respiratory motion prediction and compensation system. The actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(dx(n-1) ',d y(n-1) ',d z(n-1) ')}、The three-dimensional world coordinates of the body surface marker at time T1 in the future Future time T1 body surface marker image coordinates Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z );

[0292] Then, using actual respiratory signal sequences The respiratory motion prediction model based on LSTM network is updated online;

[0293] Using actual respiratory signal sequences and the actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) The displacement prediction model for surface markers is updated online.

[0294] Using the three-dimensional world coordinates of the body surface markers at time T1 in the future Future time T1 body surface marker image coordinates Update the camera extrinsic matrix online;

[0295] Using the image coordinates of the body surface markers at time T1 in the future Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z The tumor target center point prediction model is updated online.

[0296] Example 8:

[0297] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-7, further wherein, when training the hybrid network model based on LSTM and TCN, the parameters of the hybrid network model are optimized using the Adam optimizer after each training iteration.

[0298] Example 9:

[0299] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-8, further wherein the body surface marker position monitoring module is an electromagnetic sensor attached to the location of the marker on the patient's body surface.

[0300] Example 10:

[0301] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-9, further wherein, during the training process, the loss function used by the LSTM network and the hybrid network model based on LSTM and TCN is the mean absolute error loss.

[0302] Example 11:

[0303] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of Examples 1-10, further wherein the camera extrinsic parameter matrix is ​​obtained through calibration.

[0304] The calibration steps are as follows:

[0305] 1) Prepare the calibration plate.

[0306] 2) Acquire images.

[0307] 3) Detect feature points in the image.

[0308] 4) Calibrate the camera intrinsic parameter matrix and distortion coefficients.

[0309] 5) Solve for the extrinsic parameter matrix using the Perspective-n-Point algorithm.

[0310] 6) With the goal of minimizing reprojection error, jointly optimize the intrinsic and extrinsic parameters of all images.

[0311] Example 12:

[0312] A method and system for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, with the same technical content as any one of embodiments 1-11, further wherein the loss function of the tumor target center point prediction model is the mean absolute error function or the mean square error function.

Claims

1. A method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy, characterized in that, Real-time respiratory motion prediction is achieved through a real-time respiratory motion prediction and compensation system; The real-time respiratory motion prediction and compensation system includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate transformation module, a tumor target center point prediction module, and a tumor target visualization module. The steps to achieve real-time respiratory motion prediction include: 1) The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed. t And transmit it to the respiratory motion prediction module; 2) The respiratory motion prediction module stores a respiratory motion prediction model based on an LSTM network; The respiratory motion prediction module is based on the patient's real-time respiratory signal A. t Construct a respiratory signal sequence with respiratory signals from the past T0 time period. Then, the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the respiratory signal sequence for the future T1 time period. and respiratory state; the respiratory state is exhalation or inhalation; The respiratory motion prediction module will generate the respiratory signal sequence for the future T1 time period. The respiratory status is transmitted to the body surface marker displacement prediction module, and the respiratory status is transmitted to the tumor target center point prediction module. 3) The body surface marker position monitoring module acquires the real-time three-dimensional world coordinates (X, Y, F, Z) of the body surface markers. t ,Y t Z t The data is then transmitted to the body surface marker displacement prediction module. The body surface marker displacement prediction module stores a body surface marker prediction model. The body surface marker displacement prediction module will use the patient's respiratory signal sequence at the future T1 time period. The respiratory state is input into the displacement prediction model of the body surface markers to obtain the three-dimensional displacement change sequence d={(d x1 ,d y1 ,d z1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) The three-dimensional displacement change of the surface marker point at time T1 is determined based on the three-dimensional displacement change sequence d. 4) The surface marker displacement prediction module combines the real-time three-dimensional world coordinates and three-dimensional displacement change (d) of the surface markers. x ,d y ,d z Using the three-dimensional displacement change sequence d, the three-dimensional world coordinates of the surface marker points at time T1 in the future are determined. And the three-dimensional world coordinate sequence of the body surface markers in the future time period T1, thereby constructing the three-dimensional world coordinate prediction sequence F of the body surface markers, that is: 5) The body surface marker point coordinate transformation module stores the camera intrinsic parameter matrix and extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector; The body surface marker coordinate transformation module converts the three-dimensional world coordinate prediction sequence F of the body surface markers into the image coordinate sequence F' of the body surface markers in the future T1 time period based on the camera intrinsic and extrinsic parameter matrices, and transmits it to the tumor target center point prediction module. The image coordinate sequence F' of the body surface marker points in the future time period T1 is shown below: In the formula, u and v are image coordinates; s is the depth information of the body surface marker point; 6) The tumor target center point prediction module stores tumor target center point prediction models under different respiratory states; The tumor target center point prediction module calls the corresponding tumor target center point prediction model based on the patient's respiratory status, and uses the called tumor target center point prediction model to process the image coordinate sequence F' of the body surface marker points at the future time T1, to obtain the predicted image coordinates (Q) of the tumor target center point at the future time T1. x Q y Q z );Q z For depth information of the center point of the tumor target area; 7) The tumor target visualization module will predict the image coordinates (Q) of the center point of the tumor target at time T1 in the future. x Q y Q z Visualization; 8) After T1 time, obtain the actual respiratory signal sequence for the future T1 time period. The actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}、The three-dimensional world coordinates of the body surface marker at time T1 in the future Future time T1 body surface marker image coordinates Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z ); Then, using actual respiratory signal sequences The respiratory motion prediction model based on LSTM network is updated online; Using actual respiratory signal sequences and the actual three-dimensional displacement change sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) The displacement prediction model for surface markers is updated online. Using the three-dimensional world coordinates of the body surface markers at time T1 in the future Future time T1 body surface marker image coordinates Update the camera extrinsic matrix online; Using the image coordinates of the body surface markers at time T1 in the future Predicted image coordinates of the tumor target center point at time T1 (Q x Q y Q z The tumor target center point prediction model is updated online. The steps for constructing the body surface marker prediction model are as follows: a1) Monitor the patient's respiratory signal within time Tmax, and divide the respiratory signal into multiple respiratory signal sequences of duration T1 according to a preset step size Δt, and construct a respiratory fluctuation matrix B, i.e.: In the formula, For respiratory fluctuations The start and end times; For respiratory fluctuations Corresponding respiratory state; respiratory signal sequence At the start and end of each respiratory fluctuation, the real-time three-dimensional world coordinates (X, Y, Z) of the surface markers are obtained using the surface marker position monitoring module, and the three-dimensional displacement change sequence D = {(D x1 D y1 D z1 ),(D x2 D y2 D z2 ),...,(D x(n-1) D y(n-1) D z(n-1) )}; a2) Repeat step a1) to obtain multiple sets of respiratory fluctuation-three-dimensional displacement change sample pairs, thereby constructing a training sample set C for respiratory fluctuation-three-dimensional displacement change, i.e.: In the formula, This is a respiratory signal sequence; In a breathing state; D EG ={(D EGx1 D EGy1 D EGz1 ),(D EGx2 D EGy2 D EGz2 ),...,(D EGx(n-1) D EGy(n-1) D EGz(n-1) )} is a respiratory signal sequence The corresponding sequence of three-dimensional displacement changes at body surface markers; a3) Construct a hybrid network model based on LSTM and TCN, including an input layer, a TCN module, a bidirectional LSTM module, a multi-head attention mechanism module, and an output layer; The input layer takes a respiratory signal sequence as input. The TCN module includes at least four stacked dilated causal convolutional layers for extracting local morphological and periodic features of respiratory fluctuations. The bidirectional LSTM module includes two stacked LSTM layers, used to construct the dependency between respiratory signals and the three-dimensional displacement changes of surface markers. The multi-head attention mechanism module is used to enhance the influence weight of respiratory peaks and troughs on the three-dimensional displacement changes of body surface markers. The multi-head attention mechanism MultiHead(Q,K,V) is as follows: MultiHead(Q,K,V)=Concat(head1,...,head h ) (5) head i =Attention(QW i Q ,KW i K ,VW i V ) (6) In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; W is the scaling factor. i Q W i K W i V The projection matrix is ​​represented by `Q`; `Attention(Q,K,V)` represents the attention weights; the superscript `T` indicates transpose; `head` represents the projected matrix. i This represents the i-th attention computation unit; The output layer is used to output the three-dimensional displacement change sequence of the body surface marker points; a4) The hybrid network model based on LSTM and TCN was trained and validated using the training sample set C of respiratory fluctuation-three-dimensional displacement change to obtain the body surface marker prediction model. The body surface marker coordinate transformation module will... 3D world coordinates of the body surface marker at time Transform into the future Image coordinates of surface markers at time The steps include: b1) Convert the three-dimensional world coordinates Convert to camera coordinates, i.e.: In the formula, R1 and R2 are the camera coordinates; R1 and R2 are the rotation matrix and translation vector, respectively. The rotation matrix R1 is shown below: b2) Set camera coordinates Convert to image coordinates Right now: In the formula, depth information The intrinsic parameter matrix K is shown below: In the formula, f x f y c is the focal length; x c y The coordinates of the image center point; The steps for constructing a tumor target center point prediction model are as follows: c1) Under the same respiratory state, acquire images of surface markers and in vivo tumor target areas at time T1; c2) Extract the image coordinate sequence of the surface markers at time T1 and the image coordinates of the center point of the tumor target area in the body at time T1, and construct the mapping relationship; c3) Repeat steps c1)-c2) to obtain the tumor target center point prediction training sample set M, i.e.: In the formula, The coordinates of the marked points on the body surface; For depth information of surface markers; (M 2mx M 2my ) represents the coordinates of the center point of the tumor target area in vivo; M 2mz For depth information of the center point of the tumor target area in vivo; M 1m A sequence of image coordinates for marker points on the body surface; c3) Construct a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, gated activation layers, skip connection layers, and an output layer; Among them, the dilated causal convolutional layer is used to extract the image coordinate sequence features of the body surface markers during time T1; At time step t, the output of the dilated causal convolutional layer As shown below: In the formula, ω l d is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolutional layers; x t-d·l To expand the input of the causal convolutional layer; The output of the residual block is shown below: E'=w2σ(w1x')+w3x'(14) In the formula, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are the residual block parameters; σ is the ReLU activation function; w3 is the linear transformation function, used to ensure that the dimensions of two adjacent residual blocks are consistent; The skip connection layer transmits information from different layers to the output layer; The output layer is used to output the image coordinates of the center point of the tumor target area in vivo at time T1. c4) The bidirectional causal convolution model is trained using the tumor target center point prediction training sample set to obtain the tumor target center point prediction model.

2. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that: Respiratory signals are collected using a laser detector or a binocular camera.

3. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that: The value range of T1 is [0.5s, 2s].

4. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that: The respiratory motion prediction model includes multiple LSTM layers; During the forward propagation of the LSTM network, the output h of each LSTM unit t As shown below: f t =σ(W f ·[h t-1 ,x t ]+b f )(15) i t =σ(W i ·[h t-1 ,x t ]+b i )(16) the t =σ(W o ·[h t-1 ,x t ]+b o ) (19) h t = no t *fish(C) t ) (20) In the formula, x t This is the current input, h t-1 It is the output of the previous time step, f t i t o t These are the outputs of the forget gate, input gate, and output gate, respectively. It is a candidate cell state, C t It is a cellular state, W f W i W C W o and b f b i b C b o These are the weights and biases that the network needs to learn; σ is the activation function; C t-1 It represents the cell state at the previous moment.

5. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that, The respiratory motion prediction model based on LSTM network is constructed through the following steps: d1) Collect T data from patients on the treatment bed max The respiratory signals are collected over a period of time and then divided into multiple input-output sample pairs to construct the training sample set for the LSTM network. Each input sample has a time length of T0+1, and the output sample is the respiratory signal at time T after the input sample. d2) Train the LSTM network using the LSTM network training sample set to obtain the respiratory motion prediction model based on the LSTM network; During training, the weights and bias parameters in the LSTM network are updated using gradient descent. The updated parameters are shown below: In the formula, W′ and b′ are the updated weights and biases; α is the learning rate; L is the loss function; and W and b are the weights and biases before the update.

6. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that, When training a hybrid network model based on LSTM and TCN, the Adam optimizer is used to optimize the parameters of the hybrid network model after each training iteration.

7. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that, The body surface marker location monitoring module is an electromagnetic sensor that is attached to the location of the marker on the patient's body surface.

8. The method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that, During training, the loss function used for LSTM networks and hybrid network models based on LSTM and TCN is the mean absolute error loss.

9. A method for real-time respiratory motion prediction and compensation for thoracic and abdominal radiotherapy according to claim 1, characterized in that, The camera extrinsic matrix is ​​obtained through calibration.

Images