Real-time respiratory movement prediction and compensation method for thoracic and abdominal radiotherapy

Abstract

The invention discloses a real-time respiratory movement prediction and compensation method for thoracic and abdominal radiotherapy. A respiratory signal acquisition module, a respiratory movement prediction module, a body surface mark point position monitoring module, a body surface mark point displacement prediction module, a body surface mark point coordinate conversion module, a tumor target center point prediction module and a tumor target visualization module are included. The invention discloses a respiratory movement prediction system based on a causal convolutional network, and the system achieves the precise prediction of the position of a tumor target region in the future by collecting a respiratory waveform signal and a body surface mark point movement track of a patient in real time, extracting a periodic movement mode, and predicting the three-dimensional displacement of the tumor target region in 0.5-2 seconds in the future.

Description

Technical Field

[0001] The invention relates to the field of medical technology, and in particular to a real-time respiratory motion prediction and compensation method for chest and abdomen radiotherapy. Background Art

[0002] At present, the main method for treating lung cancer is stereotactic radiotherapy, but the respiratory movement of the human body seriously affects the accuracy of radiotherapy. In order to reduce the impact of respiratory movement, the most effective method is the real-time tracking technology of respiratory movement. This technology establishes a correlation model between the tumor in the body and the body surface markers, and uses a prediction algorithm to obtain the future movement information of the tumor according to the movement of the markers, so as to adjust the radiation beam in real time to ensure the relative stillness of the radiation and the tumor, so as to achieve the purpose of real-time tracking. The prediction algorithm is used to compensate for the time delay of the stereotactic radiotherapy system and predict in advance the position that the tumor will reach in the future.

[0003] However, existing algorithms do not consider the impact of continuous respiratory fluctuations on the position changes of tumors in the body and body surface markers. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time respiratory motion prediction and compensation method for chest and abdomen radiotherapy, which realizes real-time respiratory motion prediction through a real-time respiratory motion prediction and compensation system;

[0005] The real-time respiratory motion prediction and compensation system includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate conversion module, a tumor target area center point prediction module, and a tumor target area visualization module;

[0006] The steps to achieve real-time respiratory motion prediction include:

[0007] 1) The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed t , and transmitted to the respiratory motion prediction module;

[0008] 2) The respiratory movement prediction module stores a respiratory movement prediction model based on an LSTM network;

[0009] The respiratory motion prediction module is based on the patient's real-time respiratory signal A t and past T 0 Respiratory signals of different time periods are used to construct respiratory signal sequences Then the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the future T 1 Respiratory signal sequence of time period and a breathing state; the breathing state being exhalation or inhalation;

[0010] The respiratory motion prediction module will 1 Respiratory signal sequence of time period and the respiratory state are transmitted to the body surface marker displacement prediction module, and the respiratory state is transmitted to the tumor target area center point prediction module;

[0011] 3) The body surface marker position monitoring module obtains the real-time three-dimensional world coordinates (X t ,Y t ,Z t ) and transmitted to the body surface marker displacement prediction module;

[0012] The body surface marker point displacement prediction module stores a body surface marker point prediction model;

[0013] The body surface marker displacement prediction module will predict the patient's 1 Respiratory signal sequence of time period and respiratory status are input into the body surface marker displacement prediction model to obtain the 1 The three-dimensional displacement variation sequence of the time period d={(d x1 ,d y1 ,d z1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) )}, and determine the future T according to the three-dimensional displacement change sequence d 1 The change in the three-dimensional displacement of the body surface marker at the moment

[0014] 4) The body surface marker displacement prediction module combines the real-time 3D world coordinates and 3D displacement change (d x ,d y ,d z ) and the three-dimensional displacement variation sequence d, determine the future T 1 The three-dimensional world coordinates of the body surface marker at this moment And in the future 1 The three-dimensional world coordinate sequence of the body surface marker points in the time period is constructed to construct the three-dimensional world coordinate prediction sequence F of the body surface marker points, that is:

[0015]

[0016] 5) The body surface marker coordinate conversion module stores a camera intrinsic parameter matrix and an extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0017] The body surface marker point coordinate conversion module converts the predicted sequence F of the three-dimensional world coordinates of the body surface marker points into the future T 1 The image coordinate sequence F' of the body surface marker points in the time period is transmitted to the tumor target area center point prediction module;

[0018] Future T 1 The image coordinate sequence F' of the body surface marker points of the time period is as follows:

[0019]

[0020] Where u and v are image coordinates; s is the depth information of the body surface marker point;

[0021] 6) The tumor target area center point prediction module stores tumor target area center point prediction models under different respiratory states;

[0022] The tumor target area center point prediction module calls the corresponding tumor target area center point prediction model according to the patient's respiratory state, and uses the called tumor target area center point prediction model to predict the future T 1 The image coordinate sequence F' of the body surface marker points in the time period is processed to obtain the future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );Q z Depth information of the center point of the tumor target area;

[0023] 7) The tumor target area visualization module will be used in the future 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Visualization;

[0024] 8)T 1 After time, get the future T 1 The actual breathing signal sequence of the time period Future T 1 The actual three-dimensional displacement variation sequence of the time period d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}, future T 1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );

[0025] Then, using the actual breathing signal sequence Online update of the respiratory motion prediction model based on LSTM network;

[0026] Using actual breathing signal sequence and the actual three-dimensional displacement variation sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}Online update of the displacement prediction model of body surface marker points;

[0027] Using the future 1 3D world coordinates of body surface markers at each moment Future T 1 Image coordinates of body surface marker points at each moment Update the camera extrinsic matrix online;

[0028] Using the future 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Online update of the tumor target center prediction model.

[0029] The steps for constructing the body surface marker prediction model are as follows:

[0030] a1) Monitor the patient's respiratory signal within the Tmax time, and divide the respiratory signal into multiple time periods T according to the preset step length Δt 1 The respiratory signal sequence is constructed and the respiratory fluctuation matrix B is constructed, namely:

[0031]

[0032] In the formula, For respiratory fluctuations The start and end time of For respiratory fluctuations Corresponding breathing state; breathing signal sequence

[0033] At the start and end of each respiratory fluctuation, the body surface marker position monitoring module is used to obtain the real-time three-dimensional world coordinates (X, Y, Z) of the body surface marker point, and the three-dimensional displacement change sequence D corresponding to each respiratory signal sequence is determined. x1 ,D y1 ,D z1 ),(D x2 ,D y2 ,D z2 ),...,(D x(n-1) ,D y(n-1) ,D z(n-1) )};

[0034] a2) Repeat step a1) to obtain multiple groups of respiratory fluctuation-three-dimensional displacement variation sample pairs, thereby constructing a respiratory fluctuation-three-dimensional displacement variation training sample set C, that is:

[0035]

[0036] In the formula, is the breathing signal sequence; For breathing state; D EG ={(D EGx1 ,D EGy1 ,D EGz1 ),(D EGx2 ,D EGy2 ,D EGz2 ),...,(D EGx(n-1) ,D EGy(n-1) ,D EGz(n-1) )} is the breathing signal sequence The corresponding three-dimensional displacement change sequence of body surface marker points;

[0037] a3) Build a hybrid network model based on LSTM and TCN, including input layer, TCN module, bidirectional LSTM module, multi-head attention mechanism module and output layer;

[0038] The input layer takes the respiratory signal sequence as input;

[0039] The TCN module includes at least four stacked atrous causal convolutional layers for extracting local morphological features and periodic features of respiratory fluctuations;

[0040] The bidirectional LSTM module includes two stacked LSTM layers, which are used to construct a dependency relationship between the respiratory signal and the three-dimensional displacement change of the body surface markers;

[0041] The multi-head attention mechanism module is used to strengthen the weight of the influence of respiratory peaks and valleys on the three-dimensional displacement change of body surface markers;

[0042] The multi-head attention mechanism MultiHead(Q,K,V) is as follows:

[0043] MultiHead(Q,K,V)=Concat(head 1 ,...,head h )(5)

[0044] head i =Attention(QW i Q ,KW i K ,VW i V )(6)

[0045]

[0046] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix respectively; is the scaling factor; W i Q , W i K , W i V represents the projection matrix; Attention(Q,K,V) represents the attention weight; the superscript T represents transposition; head i represents the i-th attention calculation unit;

[0047] The output layer is used to output a sequence of three-dimensional displacement changes of body surface marker points;

[0048] a4) using the respiratory fluctuation-three-dimensional displacement variation training sample set C to train and verify the hybrid network model based on LSTM and TCN, and obtain a body surface marker prediction model;

[0049] The body surface marker coordinate conversion module will be The three-dimensional world coordinates of the body surface marker at this moment Transform to the future Image coordinates of body surface marker points at each moment The steps include:

[0050] b1) The three-dimensional world coordinates Convert to camera coordinates, that is:

[0051]

[0052] In the formula, is the camera coordinate; R 1 , R 2 is the rotation matrix (indicating the camera posture) and the translation vector (indicating the camera position);

[0053] Among them, the rotation matrix R 1 As shown below:

[0054]

[0055] b2) The camera coordinates Convert to image coordinates Right now:

[0056]

[0057] In the formula, the depth information

[0058] Among them, the internal parameter matrix K is as follows:

[0059]

[0060] In the formula, f x 、f y is the focal length; c x 、c y is the coordinate of the center point of the image;

[0061] The steps for constructing the tumor target center prediction model are as follows:

[0062] c1) Under the same breathing state, obtain 1 Time period body surface marker points and in vivo tumor target area images;

[0063] c2) Extraction of T 1 The image coordinate sequence of the body surface marker points during the time period and the T 1 The image coordinates of the center point of the tumor target area in the body at all times and construct a mapping relationship;

[0064] c3) Repeat steps c1) to c2) to obtain a tumor target area center prediction training sample set M, namely:

[0065]

[0066] In the formula, is the image coordinates of the body surface marker points; is the depth information of the body surface marker point; (M 2mx ,M 2my ) is the image coordinate of the center point of the tumor target area in vivo; M 2mz M is the depth information of the center point of the tumor target area in vivo; 1m is a sequence of image coordinates of body surface marker points;

[0067] c3) constructing a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, a gated activation layer, a skip connection layer, and an output layer;

[0068] Among them, the dilated causal convolutional layer is used to extract T 1 Image coordinate sequence characteristics of body surface marker points during a certain period of time;

[0069] At time step t, the output of the dilated causal convolutional layer is As shown below:

[0070]

[0071] In the formula, ω l is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolution layers; x t-d·l The input of the dilated causal convolutional layer;

[0072] The output of the residual block is as follows:

[0073] E'=w 2 σ(w 1 x')+w 3 x'(14)

[0074] Where E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w 1 、w 2 is the residual block parameter; σ is the ReLU activation function; w 3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are consistent;

[0075] The skip connection layer transmits information from different layers to the output layer;

[0076] The output layer is used to output the T 1 The image coordinates of the center point of the tumor target area in the body at that moment;

[0077] c4) The bidirectional causal convolution model is trained using the tumor target area center prediction training sample set to obtain a tumor target area center prediction model.

[0078] Furthermore, the breathing signal is collected by a laser detector or a binocular camera.

[0079] Further, T 1 The value range is [0.5s, 2s].

[0080] Further, the respiratory motion prediction model includes multiple LSTM layers;

[0081] During the forward propagation of the LSTM network, the output h of each LSTM unit ist As shown below:

[0082] f t =σ(W f ·[h t-1 ,x t ]+b f )(15)

[0083] i t =σ(W i ·[h t-1 ,x t ]+b i )(16)

[0084]

[0085] o t =σ(W o ·[h t-1 ,x t ]+b o )(19)

[0086] h t =o t *tanh(C t )(20)

[0087] In the formula, x t is the current input, h t-1 is the output at the previous moment, f t 、i t , o t are the outputs of the forget gate, input gate, and output gate respectively. is the candidate cell state, C t is the cell state, W f , W i , W C , W o and b f , b i , b C , b o are the weights and bias parameters that the network needs to learn; σ is the activation function; C t-1 is the cell state at the previous moment.

[0088] Furthermore, the respiratory motion prediction model based on LSTM network is constructed through the following steps:

[0089] d1) Collect T of the patient on the treatment bed max The respiratory signal of the time period is divided into multiple input-output sample pairs to construct the LSTM network training sample set; the time length of each input sample is T 0+1; the output sample is the breathing signal at time T after the input sample;

[0090] d2) using the LSTM network training sample set to train the LSTM network and obtain a respiratory motion prediction model of the LSTM network;

[0091] During the training process, the weights and bias parameters in the LSTM network are updated using the gradient descent method. The updated parameters are as follows:

[0092]

[0093] Where W′, b′ are the updated weights and biases; α is the learning rate; L is the loss function; W, b are the weights and biases before the update.

[0094] Furthermore, when the hybrid network model based on LSTM and TCN is trained, the Adam optimizer is used to optimize the parameters of the hybrid network model after each training.

[0095] Furthermore, the body surface marker position monitoring module is an electromagnetic sensor attached to the location of the patient's body surface marker.

[0096] Furthermore, during the training process, the loss function used by the LSTM network and the hybrid network model based on LSTM and TCN is the mean absolute error loss.

[0097] Furthermore, the camera extrinsic matrix is ​​obtained through calibration.

[0098] The technical effect of the present invention is unquestionable. The present invention discloses a respiratory motion prediction system based on a causal convolutional network. The present invention collects the patient's respiratory waveform signal and the motion trajectory of the body surface markers in real time, extracts the periodic motion pattern, predicts the three-dimensional displacement of the tumor target area in the next 0.5-2 seconds, and realizes the accurate prediction of the future tumor target area position. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 It is a schematic diagram of the system module;

[0100] Figure 2 The figure is a flow chart of the method. DETAILED DESCRIPTION

[0101] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0102] Embodiment 1:

[0103] See also Figure 1-Figure 2 ,This method realizes real-time respiratory motion prediction through a real-time respiratory motion prediction and compensation system;

[0104] The real-time respiratory motion prediction and compensation system includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate conversion module, a tumor target area center point prediction module, and a tumor target area visualization module;

[0105] The steps to achieve real-time respiratory motion prediction include:

[0106] 1) The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed t , and transmitted to the respiratory motion prediction module;

[0107] 2) The respiratory movement prediction module stores a respiratory movement prediction model based on an LSTM network;

[0108] The respiratory motion prediction module is based on the patient's real-time respiratory signal A t and past T 0 Respiratory signals of different time periods are used to construct respiratory signal sequences Then the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the future T 1 Respiratory signal sequence of time period and a breathing state; the breathing state being exhalation or inhalation;

[0109] The respiratory motion prediction module will 1 Respiratory signal sequence of time period and the respiratory state are transmitted to the body surface marker displacement prediction module, and the respiratory state is transmitted to the tumor target area center point prediction module;

[0110] 3) The body surface marker position monitoring module obtains the real-time three-dimensional world coordinates (X t ,Y t ,Z t ) and transmitted to the body surface marker displacement prediction module;

[0111] The body surface marker point displacement prediction module stores a body surface marker point prediction model;

[0112] The body surface marker displacement prediction module will predict the patient's 1 Respiratory signal sequence of time period and respiratory status are input into the body surface marker displacement prediction model to obtain the 1 The three-dimensional displacement variation sequence of the time period d={(d x1 ,d y1 ,dz1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) )}, and determine the future T according to the three-dimensional displacement change sequence d 1 The change in the three-dimensional displacement of the body surface marker at the moment

[0113] 4) The body surface marker displacement prediction module combines the real-time 3D world coordinates and 3D displacement change (d x ,d y ,d z ) and the three-dimensional displacement variation sequence d, determine the future T 1 The three-dimensional world coordinates of the body surface marker at this moment And in the future 1 The three-dimensional world coordinate sequence of the body surface marker points in the time period is constructed to construct the three-dimensional world coordinate prediction sequence F of the body surface marker points, that is:

[0114]

[0115] 5) The body surface marker coordinate conversion module stores a camera intrinsic parameter matrix and an extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0116] The body surface marker point coordinate conversion module converts the predicted sequence F of the three-dimensional world coordinates of the body surface marker points into the future T 1 The image coordinate sequence F' of the body surface marker points in the time period is transmitted to the tumor target area center point prediction module;

[0117] Future T 1 The image coordinate sequence F' of the body surface marker points of the time period is as follows:

[0118]

[0119] Where u and v are image coordinates; s is the depth information of the body surface marker point;

[0120] 6) The tumor target area center point prediction module stores tumor target area center point prediction models under different respiratory states;

[0121] The tumor target area center point prediction module calls the corresponding tumor target area center point prediction model according to the patient's respiratory state, and uses the called tumor target area center point prediction model to predict the future T 1 The image coordinate sequence F' of the body surface marker points in the time period is processed to obtain the future T 1The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );Q z Depth information of the center point of the tumor target area;

[0122] 7) The tumor target area visualization module will be used in the future 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Visualization;

[0123] 8)T 1 After time, get the future T 1 The actual breathing signal sequence of the time period Future T 1 The actual three-dimensional displacement variation sequence of the time period d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}, future T 1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );

[0124] Then, using the actual breathing signal sequence Online update of the respiratory motion prediction model based on LSTM network;

[0125] Using actual breathing signal sequence and the actual three-dimensional displacement variation sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}Online update of the displacement prediction model of body surface marker points;

[0126] Using the future1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1 Image coordinates of body surface marker points at each moment Update the camera extrinsic matrix online;

[0127] Using the future 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Online update of the tumor target center prediction model.

[0128] The steps for constructing the body surface marker prediction model are as follows:

[0129] a1) Monitor the patient's respiratory signal within the Tmax time, and divide the respiratory signal into multiple time periods T according to the preset step length Δt 1 The respiratory signal sequence is constructed and the respiratory fluctuation matrix B is constructed, namely:

[0130]

[0131] In the formula, For respiratory fluctuations The start and end time of For respiratory fluctuations Corresponding breathing state; breathing signal sequence

[0132] At the start and end of each respiratory fluctuation, the body surface marker position monitoring module is used to obtain the real-time three-dimensional world coordinates (X, Y, Z) of the body surface marker point, and the three-dimensional displacement change sequence D corresponding to each respiratory signal sequence is determined. x1 ,D y1 ,D z1 ),(D x2 ,D y2 ,D z2 ),...,(D x(n-1) ,D y(n-1) ,D z(n-1) )};

[0133] a2) Repeat step a1) to obtain multiple groups of respiratory fluctuation-three-dimensional displacement variation sample pairs, thereby constructing a respiratory fluctuation-three-dimensional displacement variation training sample set C, that is:

[0134]

[0135] In the formula, is the breathing signal sequence; For breathing state; D EG ={(D EGx1 ,D EGy1 ,D EGz1 ),(D EGx2 ,D EGy2 ,D EGz2 ),...,(D EGx(n-1) ,D EGy(n-1) ,D EGz(n-1) )} is the breathing signal sequence The corresponding three-dimensional displacement change sequence of body surface marker points;

[0136] a3) Build a hybrid network model based on LSTM and TCN, including input layer, TCN module, bidirectional LSTM module, multi-head attention mechanism module and output layer;

[0137] The input layer takes the respiratory signal sequence as input;

[0138] The TCN module includes at least four stacked atrous causal convolutional layers for extracting local morphological features and periodic features of respiratory fluctuations;

[0139] The bidirectional LSTM module includes two stacked LSTM layers, which are used to construct a dependency relationship between the respiratory signal and the three-dimensional displacement change of the body surface markers;

[0140] The multi-head attention mechanism module is used to strengthen the weight of the influence of respiratory peaks and valleys on the three-dimensional displacement change of body surface markers;

[0141] The multi-head attention mechanism MultiHead(Q,K,V) is as follows:

[0142] MultiHead(Q,K,V)=Concat(head 1 ,...,head h )(5)

[0143] head i =Attention(QW i Q ,KW i K ,VW i V )(6)

[0144]

[0145] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix respectively; is the scaling factor; W i Q , Wi K , W i V represents the projection matrix; Attention(Q,K,V) represents the attention weight; the superscript T represents transposition; head i represents the i-th attention calculation unit;

[0146] The output layer is used to output a sequence of three-dimensional displacement changes of body surface marker points;

[0147] a4) using the respiratory fluctuation-three-dimensional displacement variation training sample set C to train and verify the hybrid network model based on LSTM and TCN, and obtain a body surface marker prediction model;

[0148] The body surface marker coordinate conversion module will be The three-dimensional world coordinates of the body surface marker at this moment Transform to the future Image coordinates of body surface marker points at each moment The steps include:

[0149] b1) The three-dimensional world coordinates Convert to camera coordinates, that is:

[0150]

[0151] In the formula, is the camera coordinate; R 1 , R 2 is the rotation matrix (indicating the camera posture) and the translation vector (indicating the camera position);

[0152] Among them, the rotation matrix R 1 As shown below:

[0153]

[0154] b2) The camera coordinates Convert to image coordinates Right now:

[0155]

[0156] In the formula, the depth information

[0157] Among them, the internal parameter matrix K is as follows:

[0158]

[0159] In the formula, f x 、f y is the focal length; c x 、cy is the coordinate of the center point of the image;

[0160] The steps for constructing the tumor target center prediction model are as follows:

[0161] c1) Under the same breathing state, obtain 1 Time period body surface marker points and in vivo tumor target area images;

[0162] c2) Extraction of T 1 The image coordinate sequence of the body surface marker points during the time period and the T 1 The image coordinates of the center point of the tumor target area in the body at all times and construct a mapping relationship;

[0163] c3) Repeat steps c1) to c2) to obtain a tumor target area center prediction training sample set M, namely:

[0164]

[0165] In the formula, is the image coordinates of the body surface marker points; is the depth information of the body surface marker point; (M 2mx ,M 2my ) is the image coordinate of the center point of the tumor target area in vivo; M 2mz M is the depth information of the center point of the tumor target area in vivo; 1m is a sequence of image coordinates of body surface marker points;

[0166] c3) constructing a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, a gated activation layer, a skip connection layer, and an output layer;

[0167] Among them, the dilated causal convolutional layer is used to extract T 1 Image coordinate sequence characteristics of body surface marker points during a certain period of time;

[0168] At time step t, the output of the dilated causal convolutional layer is As shown below:

[0169]

[0170] In the formula, ω l is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolution layers; x t-d·l The input of the dilated causal convolutional layer;

[0171] The output of the residual block is as follows:

[0172] E'=w 2 σ(w 1 x')+w 3 x'(14)

[0173] Where E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w 1 、w 2 is the residual block parameter; σ is the ReLU activation function; w 3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are consistent;

[0174] The skip connection layer transmits information from different layers to the output layer;

[0175] The output layer is used to output the T 1 The image coordinates of the center point of the tumor target area in the body at that moment;

[0176] c4) The bidirectional causal convolution model is trained using the tumor target area center prediction training sample set to obtain a tumor target area center prediction model.

[0177] The breathing signal is collected by a laser detector or a binocular camera.

[0178] T 1 The value range is [0.5s, 2s].

[0179] The respiratory motion prediction model includes multiple LSTM layers;

[0180] During the forward propagation of the LSTM network, the output h of each LSTM unit is t As shown below:

[0181] f t =σ(W f ·[h t-1 ,x t ]+b f )(15)

[0182] i t =σ(W i ·[h t-1 ,x t ]+b i )(16)

[0183]

[0184] o t =σ(W o ·[h t-1 ,x t ]+b o )(19)

[0185] h t =o t *tanh(C t )(20)

[0186] In the formula, x t is the current input, h t-1 is the output at the previous moment, f t 、i t , o t are the outputs of the forget gate, input gate, and output gate respectively. is the candidate cell state, C t is the cell state, W f , W i , W C , W o and b f 、b i 、b C 、b o are the weights and bias parameters that the network needs to learn; σ is the activation function; C t-1 is the cell state at the previous moment.

[0187] The respiratory motion prediction model based on LSTM network is constructed through the following steps:

[0188] d1) Collect T of the patient on the treatment bed max The respiratory signal of the time period is divided into multiple input-output sample pairs to construct the LSTM network training sample set; the time length of each input sample is T 0 +1; the output sample is the breathing signal at time T after the input sample;

[0189] d2) using the LSTM network training sample set to train the LSTM network and obtain a respiratory motion prediction model of the LSTM network;

[0190] During the training process, the weights and bias parameters in the LSTM network are updated using the gradient descent method. The updated parameters are as follows:

[0191]

[0192] Where W′, b′ are the updated weights and biases; α is the learning rate; L is the loss function; W, b are the weights and biases before the update.

[0193] During the operation of the real-time respiratory motion prediction and compensation system, the future T 1 The actual breathing signal sequence of the time period Future T 1 The actual three-dimensional displacement variation sequence of the time period d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2'),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}, future T 1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );

[0194] Then, using the actual breathing signal sequence Online update of the respiratory motion prediction model based on LSTM network;

[0195] Using actual breathing signal sequence and the actual three-dimensional displacement variation sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}Online update of the displacement prediction model of body surface marker points;

[0196] Using the future 1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1 Image coordinates of body surface marker points at each moment Update the camera extrinsic matrix online;

[0197] Using the future 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Online update of the tumor target center prediction model.

[0198] When training a hybrid network model based on LSTM and TCN, the Adam optimizer is used to optimize the parameters of the hybrid network model after each training.

[0199] The body surface marker position monitoring module is an electromagnetic sensor attached to the position of the patient's body surface marker.

[0200] During the training process, the loss function used by the LSTM network and the hybrid network model based on LSTM and TCN is the mean absolute error loss.

[0201] The camera extrinsic matrix is ​​obtained through calibration.

[0202] Embodiment 2:

[0203] A real-time respiratory motion prediction and compensation system for chest and abdominal radiotherapy, including a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate conversion module, a tumor target area center point prediction module, and a tumor target area visualization module;

[0204] The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed. t , and transmitted to the respiratory motion prediction module;

[0205] The respiratory movement prediction module stores a respiratory movement prediction model based on an LSTM network;

[0206] The respiratory motion prediction module is based on the patient's real-time respiratory signal A t and past T 0 Respiratory signals of different time periods are used to construct respiratory signal sequences Then the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the future T 1 Respiratory signal sequence of time period and a breathing state; the breathing state being exhalation or inhalation;

[0207] The respiratory motion prediction module will 1 Respiratory signal sequence of time period and the respiratory state are transmitted to the body surface marker displacement prediction module, and the respiratory state is transmitted to the tumor target area center point prediction module;

[0208] The body surface marker position monitoring module obtains the real-time three-dimensional world coordinates (X t ,Y t ,Z t ) and transmitted to the body surface marker displacement prediction module;

[0209] The body surface marker point displacement prediction module stores a body surface marker point prediction model;

[0210] The body surface marker displacement prediction module will predict the patient's 1 Respiratory signal sequence of time period and respiratory status are input into the body surface marker displacement prediction model to obtain the 1 The three-dimensional displacement variation sequence of the time period d={(d x1,d y1 ,d z1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) )}, and determine the future T according to the three-dimensional displacement change sequence d 1 The change in the three-dimensional displacement of the body surface marker at the moment

[0211] The body surface marker point displacement prediction module combines the real-time three-dimensional world coordinates of the body surface marker point and the three-dimensional displacement change (d x ,d y ,d z ) and the three-dimensional displacement variation sequence d, determine the future T 1 The three-dimensional world coordinates of the body surface marker at this moment And in the future 1 The three-dimensional world coordinate sequence of the body surface marker points in the time period is constructed to construct the three-dimensional world coordinate prediction sequence F of the body surface marker points, that is:

[0212]

[0213] The body surface marker coordinate conversion module stores a camera intrinsic parameter matrix and an extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0214] The body surface marker point coordinate conversion module converts the predicted sequence F of the three-dimensional world coordinates of the body surface marker points into the future T 1 The image coordinate sequence F' of the body surface marker points in the time period is transmitted to the tumor target area center point prediction module;

[0215] Future T 1 The image coordinate sequence F' of the body surface marker points of the time period is as follows:

[0216]

[0217] Where u and v are image coordinates; s is the depth information of the body surface marker point;

[0218] The tumor target area center point prediction module stores tumor target area center point prediction models under different respiratory states;

[0219] The tumor target area center point prediction module calls the corresponding tumor target area center point prediction model according to the patient's respiratory state, and uses the called tumor target area center point prediction model to predict the future T 1 The image coordinate sequence F' of the body surface marker points in the time period is processed to obtain the future T 1The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );Q z Depth information of the center point of the tumor target area;

[0220] The tumor target area visualization module will be 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Visualization;

[0221] The steps for constructing the body surface marker prediction model are as follows:

[0222] a1) Monitor the patient's respiratory signal within the Tmax time, and divide the respiratory signal into multiple time periods T according to the preset step length Δt 1 The respiratory signal sequence is constructed and the respiratory fluctuation matrix B is constructed, namely:

[0223]

[0224] In the formula, For respiratory fluctuations The start and end time of For respiratory fluctuations Corresponding breathing state; breathing signal sequence

[0225] At the start and end of each respiratory fluctuation, the body surface marker position monitoring module is used to obtain the real-time three-dimensional world coordinates (X, Y, Z) of the body surface marker point, and the three-dimensional displacement change sequence D corresponding to each respiratory signal sequence is determined. x1 ,D y1 ,D z1 ),(D x2 ,D y2 ,D z2 ),...,(D x(n-1) ,D y(n-1) ,D z(n-1) )};

[0226] a2) Repeat step a1) to obtain multiple groups of respiratory fluctuation-three-dimensional displacement variation sample pairs, thereby constructing a respiratory fluctuation-three-dimensional displacement variation training sample set C, that is:

[0227]

[0228] In the formula, is the breathing signal sequence; For breathing state; D EG ={(DEGx1 ,D EGy1 ,D EGz1 ),(D EGx2 ,D EGy2 ,D EGz2 ),...,(D EGx(n-1) ,D EGy(n-1) ,D EGz(n-1) )} is the breathing signal sequence The corresponding three-dimensional displacement change sequence of body surface marker points;

[0229] a3) Build a hybrid network model based on LSTM and TCN, including input layer, TCN module, bidirectional LSTM module, multi-head attention mechanism module and output layer;

[0230] The input layer takes the respiratory signal sequence as input;

[0231] The TCN module includes at least four stacked atrous causal convolutional layers for extracting local morphological features and periodic features of respiratory fluctuations;

[0232] The bidirectional LSTM module includes two stacked LSTM layers, which are used to construct a dependency relationship between the respiratory signal and the three-dimensional displacement change of the body surface markers;

[0233] The multi-head attention mechanism module is used to strengthen the weight of the influence of respiratory peaks and valleys on the three-dimensional displacement change of body surface markers;

[0234] The multi-head attention mechanism MultiHead(Q,K,V) is as follows:

[0235] MultiHead(Q,K,V)=Concat(head 1 ,...,head h )(5)

[0236] head i =Attention(QW i Q ,KW i K ,VW i V )(6)

[0237]

[0238] In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix respectively; is the scaling factor; W i Q , W i K , W iV represents the projection matrix; Attention(Q,K,V) represents the attention weight; the superscript T represents transposition; head i represents the i-th attention calculation unit;

[0239] The output layer is used to output a sequence of three-dimensional displacement changes of body surface marker points;

[0240] a4) using the respiratory fluctuation-three-dimensional displacement variation training sample set C to train and verify the hybrid network model based on LSTM and TCN, and obtain a body surface marker prediction model;

[0241] The body surface marker coordinate conversion module will be The three-dimensional world coordinates of the body surface marker at this moment Transform to the future Image coordinates of body surface marker points at each moment The steps include:

[0242] b1) The three-dimensional world coordinates Convert to camera coordinates, that is:

[0243]

[0244] In the formula, is the camera coordinate; R 1 , R 2 is the rotation matrix (indicating the camera posture) and the translation vector (indicating the camera position);

[0245] Among them, the rotation matrix R 1 As shown below:

[0246]

[0247] b2) The camera coordinates Convert to image coordinates Right now:

[0248]

[0249] In the formula, the depth information

[0250] Among them, the internal parameter matrix K is as follows:

[0251]

[0252] In the formula, f x 、f y is the focal length; c x 、c y is the coordinate of the center point of the image;

[0253] The steps for constructing the tumor target center prediction model are as follows:

[0254] c1) Under the same breathing state, obtain 1 Time period body surface marker points and in vivo tumor target area images;

[0255] c2) Extraction of T 1 The image coordinate sequence of the body surface marker points during the time period and the T 1 The image coordinates of the center point of the tumor target area in the body at all times and construct a mapping relationship;

[0256] c3) Repeat steps c1) to c2) to obtain a tumor target area center prediction training sample set M, namely:

[0257]

[0258] In the formula, is the image coordinates of the body surface marker points; is the depth information of the body surface marker point; (M 2mx ,M 2my ) is the image coordinate of the center point of the tumor target area in vivo; M 2mz M is the depth information of the center point of the tumor target area in vivo; 1m is a sequence of image coordinates of body surface marker points;

[0259] c3) constructing a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, a gated activation layer, a skip connection layer, and an output layer;

[0260] Among them, the dilated causal convolutional layer is used to extract T 1 Image coordinate sequence characteristics of body surface marker points during a certain period of time;

[0261] At time step t, the output of the dilated causal convolutional layer is As shown below:

[0262]

[0263] In the formula, ω l is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolution layers; x t-d·l The input of the dilated causal convolutional layer;

[0264] The output of the residual block is as follows:

[0265] E'=w 2 σ(w 1 x')+w 3 x'(14)

[0266] Where E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w 1 、w 2 is the residual block parameter; σ is the ReLU activation function; w 3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are consistent;

[0267] The skip connection layer transmits information from different layers to the output layer;

[0268] The output layer is used to output the T 1 The image coordinates of the center point of the tumor target area in the body at that moment;

[0269] c4) The bidirectional causal convolutional model is trained using the tumor target area center prediction training sample set to obtain a tumor target area center prediction model.

[0270] Embodiment 3:

[0271] A real-time respiratory motion prediction and compensation method and system for chest and abdominal radiotherapy, the technical content of which is the same as any one of Examples 1-2, and further, the respiratory signal is collected by a laser detector or a binocular camera.

[0272] Embodiment 4:

[0273] A real-time respiratory motion prediction and compensation method and system for chest and abdominal radiotherapy, the technical content is the same as any one of embodiments 1-3, further, T 1 The value range is [0.5s, 2s].

[0274] Embodiment 5:

[0275] A real-time respiratory motion prediction and compensation method and system for thoracic and abdominal radiotherapy, the technical content of which is the same as any one of Embodiments 1-4, further, the respiratory motion prediction model includes multiple LSTM layers;

[0276] During the forward propagation of the LSTM network, the output h of each LSTM unit is t As shown below:

[0277] f t =σ(W f ·[h t-1 ,x t ]+b f )(15)

[0278] i t =σ(W i ·[h t-1 ,x t ]+b i )(16)

[0279]

[0280] o t =σ(W o ·[h t-1 ,x t ]+b o )(19)

[0281] h t =o t *tanh(C t )(20)

[0282] In the formula, x t is the current input, h t-1 is the output at the previous moment, f t 、i t , o t are the outputs of the forget gate, input gate, and output gate respectively. is the candidate cell state, C t is the cell state, W f , W i , W C , W o and b f 、b i 、b C 、b o are the weights and bias parameters that the network needs to learn; σ is the activation function; C t-1 is the cell state at the previous moment.

[0283] Embodiment 6:

[0284] A real-time respiratory motion prediction and compensation method and system for thoracic and abdominal radiotherapy, the technical content is the same as any one of embodiments 1-5, further, a respiratory motion prediction model based on an LSTM network is constructed by the following steps:

[0285] d1) Collect T of the patient on the treatment bed max The respiratory signal of the time period is divided into multiple input-output sample pairs to construct the LSTM network training sample set; the time length of each input sample is T 0 +1; the output sample is the breathing signal at time T after the input sample;

[0286] d2) using the LSTM network training sample set to train the LSTM network and obtain a respiratory motion prediction model of the LSTM network;

[0287] During the training process, the weights and bias parameters in the LSTM network are updated using the gradient descent method. The updated parameters are as follows:

[0288]

[0289] Where W′, b′ are the updated weights and biases; α is the learning rate; L is the loss function; W, b are the weights and biases before the update.

[0290] Embodiment 7:

[0291] A real-time respiratory motion prediction and compensation method and system for chest and abdominal radiotherapy, the technical content is the same as any one of embodiments 1-6, further, during the operation of the real-time respiratory motion prediction and compensation system, the future T 1 The actual breathing signal sequence of the time period Future T 1 The actual three-dimensional displacement variation sequence of the time period d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}, future T 1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z );

[0292] Then, using the actual breathing signal sequence Online update of the respiratory motion prediction model based on LSTM network;

[0293] Using actual breathing signal sequence and the actual three-dimensional displacement variation sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}Online update of the displacement prediction model of body surface marker points;

[0294] Using the future 1 The three-dimensional world coordinates of the body surface marker at this moment Future T 1Image coordinates of body surface marker points at each moment Update the camera extrinsic matrix online;

[0295] Using the future 1 Image coordinates of body surface marker points at each moment Future T 1 The predicted image coordinates of the tumor target center point at the moment (Q x ,Q y ,Q z ) Online update of the tumor target center prediction model.

[0296] Embodiment 8:

[0297] A real-time respiratory motion prediction and compensation method and system for thoracic and abdominal radiotherapy, the technical content of which is the same as any one of Examples 1-7. Furthermore, when a hybrid network model based on LSTM and TCN is trained, the Adam optimizer is used to optimize the hybrid network model parameters after each training.

[0298] Embodiment 9:

[0299] A real-time respiratory motion prediction and compensation method and system for chest and abdominal radiotherapy, the technical content of which is the same as any one of Examples 1-8, and further, the body surface marker position monitoring module is an electromagnetic sensor attached to the location of the patient's body surface marker.

[0300] Embodiment 10:

[0301] A real-time respiratory motion prediction and compensation method and system for thoracic and abdominal radiotherapy, the technical content of which is the same as any one of Examples 1-9, and further, during the training process, the loss function adopted by the LSTM network and the hybrid network model based on LSTM and TCN is the mean absolute error loss.

[0302] Embodiment 11:

[0303] A real-time respiratory motion prediction and compensation method and system for chest and abdominal radiotherapy, the technical content of which is the same as any one of Examples 1-10, and further, the camera extrinsic parameter matrix is ​​obtained by calibration.

[0304] The calibration steps are as follows:

[0305] 1) Prepare the calibration plate.

[0306] 2) Collect images.

[0307] 3) Detect feature points in the image.

[0308] 4) Calibrate the camera intrinsic parameter matrix and distortion coefficients.

[0309] 5) Solve the extrinsic parameter matrix using the Perspective-n-Point algorithm.

[0310] 6) With the goal of minimizing the reprojection error, the intrinsic and extrinsic parameters of all images are jointly optimized.

[0311] Embodiment 12:

[0312] A real-time respiratory motion prediction and compensation method and system for thoracic and abdominal radiotherapy, the technical content of which is the same as any one of Examples 1-11, and further, the loss function of the tumor target area center point prediction model is a mean absolute error function or a mean square error function.

Claims

1. A real-time respiratory motion prediction and compensation method for chest and abdominal radiotherapy, characterized in that: Real-time respiratory motion prediction is achieved through real-time respiratory motion prediction and compensation system; The real-time respiratory motion prediction and compensation system includes a respiratory signal acquisition module, a respiratory motion prediction module, a body surface marker position monitoring module, a body surface marker displacement prediction module, a body surface marker coordinate conversion module, a tumor target area center point prediction module, and a tumor target area visualization module; The steps to achieve real-time respiratory motion prediction include: 1) The respiratory signal acquisition module acquires the real-time respiratory signal A of the patient on the treatment bed t , and transmitted to the respiratory motion prediction module; 2) The respiratory movement prediction module stores a respiratory movement prediction model based on an LSTM network; The respiratory motion prediction module is based on the patient's real-time respiratory signal A t And the respiratory signal of the past T0 period, construct the respiratory signal sequence Then the respiratory signal sequence A is input into the respiratory motion prediction model to obtain the respiratory signal sequence of the future T1 period and a breathing state; the breathing state being exhalation or inhalation; The respiratory motion prediction module calculates the respiratory signal sequence of the future T1 period and the respiratory state are transmitted to the body surface marker displacement prediction module, and the respiratory state is transmitted to the tumor target area center point prediction module; 3) The body surface marker position monitoring module obtains the real-time three-dimensional world coordinates (X t ,Y t ,Z t ) and transmitted to the body surface marker displacement prediction module; The body surface marker point displacement prediction module stores a body surface marker point prediction model; The body surface marker displacement prediction module calculates the patient's respiratory signal sequence in the future T1 period. and respiratory status are input into the displacement prediction model of body surface markers to obtain the three-dimensional displacement change sequence d={(d x1 ,d y1 ,d z1 ),(d x2 ,d y2 ,d z2 ),...,(d x(n-1) ,d y(n-1) ,d z(n-1) )}, and determine the three-dimensional displacement change of the body surface marker point at the future time T1 according to the three-dimensional displacement change sequence d 4) The body surface marker displacement prediction module combines the real-time 3D world coordinates and 3D displacement change (d x ,d y ,d z ) and the three-dimensional displacement change sequence d, determine the three-dimensional world coordinates of the body surface marker point at the future time T1 And the three-dimensional world coordinate sequence of the body surface marker points in the future T1 period, so as to construct the three-dimensional world coordinate prediction sequence F of the body surface marker points, that is: 5) The body surface marker coordinate conversion module stores a camera intrinsic parameter matrix and an extrinsic parameter matrix; the extrinsic parameter matrix includes a rotation matrix and a translation vector; The body surface marker point coordinate conversion module converts the body surface marker point three-dimensional world coordinate prediction sequence F into the body surface marker point image coordinate sequence F' of the future T1 period based on the camera intrinsic parameter matrix and the extrinsic parameter matrix, and transmits it to the tumor target area center point prediction module; The image coordinate sequence F' of the body surface marker points in the future T1 period is as follows: Where u and v are image coordinates; s is the depth information of the body surface marker point; 6) The tumor target area center point prediction module stores tumor target area center point prediction models under different respiratory states; The tumor target area center prediction module calls the corresponding tumor target area center prediction model according to the patient's respiratory state, and uses the called tumor target area center prediction model to process the body surface marker image coordinate sequence F' of the future T1 period to obtain the predicted image coordinates of the tumor target area center at the future T1 moment (Q x ,Q y ,Q z );Q z Depth information of the center point of the tumor target area; 7) The tumor target area visualization module predicts the image coordinates (Q x ,Q y ,Q z ) Visualization; 8) After T1 time, obtain the actual respiratory signal sequence of the future T1 period The actual three-dimensional displacement variation sequence d' in the future T1 period is x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}, the three-dimensional world coordinates of the body surface marker point at the future T1 moment Image coordinates of body surface marker points at the future time T1 The predicted image coordinates of the tumor target center at the future T1 moment (Q x ,Q y ,Q z ); Then, using the actual breathing signal sequence Online update of the respiratory motion prediction model based on LSTM network; Using actual breathing signal sequence and the actual three-dimensional displacement variation sequence d'={(d x1 ',d y1 ',d z1 '),(d x2 ',d y2 ',d z2 '),...,(d x(n-1) ',d y(n-1) ',d z(n-1) ')}Online update of the displacement prediction model of body surface marker points; Use the three-dimensional world coordinates of the body surface marker point at the future time T1 Image coordinates of body surface marker points at the future time T1 Update the camera extrinsic matrix online; Use the image coordinates of the body surface marker points at the future time T1 The predicted image coordinates of the tumor target center at the future T1 moment (Q x ,Q y ,Q z ) Online update of the tumor target center prediction model. The steps for constructing the body surface marker prediction model are as follows: a1) Monitor the patient's respiratory signal within the Tmax time, and divide the respiratory signal into multiple respiratory signal sequences of duration T1 according to the preset step length Δt, and construct the respiratory fluctuation matrix B, that is: In the formula, For respiratory fluctuations The start and end time of For respiratory fluctuations Corresponding breathing state; breathing signal sequence At the start and end of each respiratory fluctuation, the body surface marker position monitoring module is used to obtain the real-time three-dimensional world coordinates (X, Y, Z) of the body surface marker point, and the three-dimensional displacement change sequence D corresponding to each respiratory signal sequence is determined. x1 ,D y1 ,D z1 ),(D x2 ,D y2 ,D z2 ),...,(D x(n-1) ,D y(n-1) ,D z(n-1) )}; a2) Repeat step a1) to obtain multiple groups of respiratory fluctuation-three-dimensional displacement variation sample pairs, thereby constructing a respiratory fluctuation-three-dimensional displacement variation training sample set C, that is: In the formula, is the breathing signal sequence; For breathing state; D EG ={(D EGx1 ,D EGy1 ,D EGz1 ),(D EGx2 ,D EGy2 ,D EGz2 ),...,(D EGx(n-1) ,D EGy(n-1) ,D EGz(n-1) )} is the breathing signal sequence The corresponding three-dimensional displacement change sequence of body surface marker points; a3) Build a hybrid network model based on LSTM and TCN, including input layer, TCN module, bidirectional LSTM module, multi-head attention mechanism module and output layer; The input layer takes the respiratory signal sequence as input; The TCN module includes at least four stacked atrous causal convolutional layers for extracting local morphological features and periodic features of respiratory fluctuations; The bidirectional LSTM module includes two stacked LSTM layers, which are used to construct a dependency relationship between the respiratory signal and the three-dimensional displacement change of the body surface markers; The multi-head attention mechanism module is used to strengthen the weight of the influence of respiratory peaks and valleys on the three-dimensional displacement change of body surface markers; The multi-head attention mechanism MultiHead(Q,K,V) is as follows: MultiHead(Q,K,V)=Concat(head1,...,head h ) (5) head i =Attention(QW i Q ,KW i K ,VW i V ) (6) In the formula, Q, K, and V represent the query matrix, key matrix, and value matrix respectively; is the scaling factor; W i Q , W i K , W i V represents the projection matrix; Attention(Q,K,V) represents the attention weight; the superscript T represents transposition; head i represents the i-th attention calculation unit; The output layer is used to output a sequence of three-dimensional displacement changes of body surface marker points; a4) using the respiratory fluctuation-three-dimensional displacement variation training sample set C to train and verify the hybrid network model based on LSTM and TCN, and obtain a body surface marker prediction model; The body surface marker coordinate conversion module will be The three-dimensional world coordinates of the body surface marker at this moment Transform to the future Image coordinates of body surface marker points at each moment The steps include: b1) The three-dimensional world coordinates Convert to camera coordinates, that is: In the formula, is the camera coordinate; R1 and R2 are the rotation matrix and translation vector; Among them, the rotation matrix R1 is as follows: b2) The camera coordinates Convert to image coordinates Right now: In the formula, the depth information Among them, the internal parameter matrix K is as follows: In the formula, f x 、f y is the focal length; c x 、c y is the coordinate of the center point of the image; The steps for constructing the tumor target center prediction model are as follows: c1) Under the same respiratory state, obtain images of body surface markers and in vivo tumor target areas at T1 period; c2) extracting the image coordinate sequence of the body surface markers at the T1 period and the image coordinates of the center point of the tumor target area in the body at the T1 moment, and constructing a mapping relationship; c3) Repeat steps c1) to c2) to obtain a tumor target area center prediction training sample set M, namely: In the formula, is the image coordinates of the body surface marker points; is the depth information of the body surface marker point; (M 2mx ,M 2my ) is the image coordinate of the center point of the tumor target area in vivo; M 2mz M is the depth information of the center point of the tumor target area in vivo; 1m is a sequence of image coordinates of body surface marker points; c3) constructing a causal convolutional neural network model, including multiple dilated causal convolutional layers, multiple residual blocks, a gated activation layer, a skip connection layer, and an output layer; Among them, the dilated causal convolutional layer is used to extract the image coordinate sequence features of the body surface markers during the T1 period; At time step t, the output of the dilated causal convolutional layer is As shown below: In the formula, ω l is the weight of the convolution kernel; d is the dilation factor; L is the number of dilated causal convolution layers; x t-d·l The input of the dilated causal convolutional layer; The output of the residual block is as follows: E'=w2σ(w1x')+w3x'(14) Wherein, E' is the output of the residual block; x' is the output of the previous convolutional layer adjacent to the current residual block; w1 and w2 are residual block parameters; σ is the ReLU activation function; w3 is a linear transformation function used to ensure that the dimensions of two adjacent residual blocks are consistent; The skip connection layer transmits information from different layers to the output layer; The output layer is used to output the image coordinates of the center point of the tumor target area in the body at time T1; c4) The bidirectional causal convolution model is trained using the tumor target area center prediction training sample set to obtain a tumor target area center prediction model.

2. A real-time respiratory motion prediction and compensation method for chest and abdominal radiotherapy according to claim 1, characterized in that: The breathing signal is collected by a laser detector or a binocular camera.

3. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: The value range of T1 is [0.5s, 2s].

4. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: The respiratory motion prediction model includes multiple LSTM layers; During the forward propagation of the LSTM network, the output h of each LSTM unit is t As shown below: f t =σ(W f ·[h t-1 ,x t ]+b f )(15) i t =σ(W i ·[h t-1 ,x t ]+b i )(16) the t =σ(W o ·[h t-1 ,x t ]+b o ) (19) h t =o t *tanh(C t ) (20) In the formula, x t is the current input, h t-1 is the output at the previous moment, f t 、i t , o t are the outputs of the forget gate, input gate, and output gate, respectively. is the candidate cell state, C t is the cell state, W f , W i , W C , W o and b f , b i , b C , b o are the weights and bias parameters that the network needs to learn; σ is the activation function; C t-1 is the cell state at the previous moment.

5. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: The respiratory motion prediction model based on LSTM network is constructed through the following steps: d1) Collect T of the patient on the treatment bed max The respiratory signal of the time period is divided into multiple input-output sample pairs to construct the LSTM network training sample set; the time length of each input sample is T0+1; the output sample is the respiratory signal at time T after the input sample; d2) using the LSTM network training sample set to train the LSTM network and obtain a respiratory motion prediction model of the LSTM network; During the training process, the weights and bias parameters in the LSTM network are updated using the gradient descent method. The updated parameters are as follows: Where W′, b′ are the updated weights and biases; α is the learning rate; L is the loss function; W, b are the weights and biases before the update.

6. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: When training a hybrid network model based on LSTM and TCN, the Adam optimizer is used to optimize the parameters of the hybrid network model after each training.

7. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: The body surface marker position monitoring module is an electromagnetic sensor attached to the position of the patient's body surface marker.

8. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: During the training process, the loss function used by the LSTM network and the hybrid network model based on LSTM and TCN is the mean absolute error loss.

9. The method for real-time respiratory motion prediction and compensation for chest and abdominal radiotherapy according to claim 1, characterized in that: The camera extrinsic matrix is ​​obtained through calibration.

Images