A method and system for synthesizing real-time images using optical body surface motion signals

Abstract

The present invention provides a method and system for synthesizing real-time images using optical body surface motion signals, including: an acquisition unit for acquiring images of patients before treatment and real-time optical body surface data during treatment; a synthesis unit for synthesizing the acquired images of patients before treatment and real-time optical body surface data during treatment into real-time images synchronized with changes in optical body surface motion signals according to a certain mapping relationship. The scheme provided by the present invention has the following beneficial effects: the synthesized real-time image realizes accurate, real-time, non-invasive, visual dynamic tracking of moving target areas during radiotherapy and no additional dose monitoring on the existing traditional accelerator platform, and has the advantages of low cost, good compatibility and universality.

Description

Technical Field

[0001] The present invention relates to the technical field of radiotherapy equipment, and in particular to a method and system for synthesizing real-time images using optical body surface motion signals. Background Art

[0002] The rapidly increasing incidence and number of patients with malignant tumors have placed a heavy burden on my country's social and economic development and health care security. As one of the main anti-cancer methods in contemporary times, radiotherapy achieves the goal of killing tumors while protecting adjacent normal tissues by accurately delivering a lethal radiation dose to the tumor target area and using advanced technologies such as intensity modulation to form a rapid dose drop at its outer edge. Radiotherapy dose off-target is one of the main causes of tumor control failure and damage to important organs. Although image-guided radiotherapy (IGRT) has greatly improved treatment accuracy by reducing patient positioning errors, real-time monitoring of the moving target area during radiotherapy is a technical bottleneck that IGRT urgently needs to break through, and it is also a common problem that crosses multiple disciplines such as science, engineering, medicine, and information.

[0003] Respiratory motion is the main risk of off-target dose in thoracic and abdominal radiotherapy. The amplitude and frequency of respiratory motion are determined by many factors, including the patient's gender and age, mental state, related diseases, radiotherapy position fixation device, and respiratory motion management measures. There are complex individual differences and uncertainties in its pattern. The influence on the traction deformation and movement trajectory of the tumor is the main threat to the accurate delivery of thoracic and abdominal radiotherapy doses. In severe cases, insufficient irradiation of the target area may lead to recurrence and metastasis, or normal organs may be mistakenly entered into the high-dose area in the irradiation field, causing serious damage or even life-threatening. Since techniques such as breath-hold therapy and active breathing control (ABC) require high patient compliance and tolerance, the free breathing mode is still the first choice for more radiotherapy patients. In the preparation stage of radiotherapy, evaluating the motion amplitude of the target area and expanding the irradiation range by extending the clinical target volume (CTV) to a certain boundary to form an internal target volume (ITV) is a common method to avoid missed irradiation of the moving target area in clinical practice. However, the larger the motion amplitude, the more normal tissues will be included in the exponentially growing ITV volume. The radiation damage caused by irradiating adjacent normal tissues as targets is not only detrimental to the protection of patients' quality of life, but also restricts the further increase of target prescription doses to improve efficacy.

[0004] Respiratory movement can change the position and shape of thoracic and abdominal tumors, which can cause off-target radiotherapy doses, leading to recurrence or normal organ damage. Positioning four-dimensional CT and MRI, as well as traditional four-dimensional cone-beam CT for guiding positioning before treatment, cannot reflect the real-time dynamics of the target area during treatment; external indirect signals such as respiratory gating and optical surface during treatment cannot directly display internal structures; real-time fluoroscopic imaging increases radiation risk; and MR-accelerators, which have not yet been popularized, have limitations such as price and compatibility. Therefore, the development of universal real-time imaging technology based on a standard platform has great scientific significance and clinical value.

[0005] In order to balance the contradiction between off-target risk and normal tissue protection, it is an urgent clinical need to reduce the safety margin of ITV through technical means such as image guidance, respiratory gating, and target area tracking. Real-time, non-invasive, and visual monitoring of moving target areas in the body has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The main purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and system for synthesizing real-time images using optical body surface motion signals, so as to solve the problem that respiratory movement will change the position and shape of chest and abdominal tumors, causing radiotherapy dose off-target, leading to recurrence or normal organ damage, which cannot be solved by the prior art.

[0007] One of the embodiments of the present invention provides a system for synthesizing real-time images using optical body surface motion signals, including: an acquisition unit, used to acquire images of patients before treatment and real-time optical body surface data during treatment; a synthesis unit, used to synthesize the acquired images of patients before treatment and real-time optical body surface data during treatment into real-time images synchronized with changes in optical body surface motion signals according to a certain mapping relationship.

[0008] In some embodiments, the image of the patient before treatment is a directly acquired four-dimensional image before treatment, or a four-dimensional image reconstructed from a three-dimensional image of the patient before treatment.

[0009] In some embodiments, before synthesizing the acquired images of the patient before treatment and the real-time optical body surface data during treatment into a real-time image synchronized with the change of the optical body surface motion signal according to a certain mapping relationship, the real-time optical body surface data is standardized.

[0010] In some embodiments, a deep learning network model unit having the mapping relationship is also included, which is used to obtain images of a certain phase i before treatment of the patient and optical surface data of phase j during treatment, input the deep learning network model, and output the image of phase j. The images of consecutive phases j are combined to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal.

[0011] One of the embodiments of the present invention provides a method for synthesizing real-time images using optical body surface motion signals, obtaining images of patients before treatment and real-time optical body surface data during treatment; the images of patients before treatment and real-time optical body surface data during treatment are synthesized into real-time images synchronized with changes in optical body surface motion signals according to a certain mapping relationship.

[0012] In some embodiments, the image of the patient before treatment is a directly acquired four-dimensional image before treatment, or a four-dimensional image reconstructed from a three-dimensional image of the patient before treatment.

[0013] In some embodiments, before synthesizing the acquired images of the patient before treatment and the real-time optical body surface data during treatment into a real-time image synchronized with the change of the optical body surface motion signal according to a certain mapping relationship, the real-time optical body surface data is standardized.

[0014] In some embodiments, the mapping relationship is obtained through a deep learning network model, and the image of a certain phase i before the patient's treatment and the optical surface data of phase j during the treatment are obtained, the deep learning network model is input, the image of phase j is output, and the images of consecutive phase j are combined to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal.

[0015] One of the embodiments of the present invention provides a deep learning network model training system for synthesizing real-time images using optical body surface motion signals, characterized in that it includes a deep learning network model unit for training the deep learning network model to obtain a mapping relationship between the patient's body surface and internal structure data; the deep learning network model is trained on a four-dimensional image data set, the deep learning network model input is a four-dimensional image of a certain phase i and optical body surface data or body surface external contour data of another phase j, and the deep learning network model output is an image of phase j.

[0016] One of the embodiments of the present invention provides a deep learning network model training method for synthesizing real-time images using optical body surface motion signals, characterized in that the deep learning network model is trained to obtain a mapping relationship between the patient's body surface and internal structure data; the deep learning network model is trained on a four-dimensional image data set, the deep learning network model input is a four-dimensional image of a certain phase i and optical body surface data or body surface external contour data of another phase j, and the deep learning network model output is an image of phase j.

[0017] The beneficial effects of the scheme of the present invention are as follows: The present invention uses deep learning technology to mine and analyze a large number of four-dimensional images of patients before treatment, such as: 4D-CBCT, 4D-CT, 4D-MR and other in vivo motion images and the characteristics of the optical surface respiratory motion signals during treatment are associated and a dynamic mapping relationship is established, and then the cone beam CT before treatment and the synchronous optical surface motion signals during treatment are used to synthesize real-time four-dimensional cone beam CT. With motion target tracking as the main goal, a dynamic mapping model of three- / four-dimensional image anatomical information and optical surface motion signal characteristics is established based on deep learning technology and multimodal medical big data. Realize real-time, non-invasive, and dynamic visualization monitoring of the internal structure of the in vivo motion target. Provide a real-time monitoring solution for motion target areas with lower economic cost, wider patient applicability, and better compatibility with existing sites and traditional quality control equipment, improve the accuracy and timeliness of motion target area radiotherapy, and reduce the risk of tumor recurrence and metastasis and normal organ radiation damage caused by off-target. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention will be further described in the form of exemplary embodiments, which will be described in detail by way of the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0019] Figure 1 is a schematic diagram of a system for synthesizing a real-time image using an optical body surface motion signal according to some embodiments of the present invention;

[0020] Figure 2 is a schematic diagram of a method for synthesizing a real-time image using an optical body surface motion signal according to some embodiments of the present invention;

[0021] Figure 3 is a two-dimensional height map of an optical body surface signal according to some embodiments of the present invention;

[0022] Figure 4 is a three-dimensional body surface Mask image of an optical body surface signal according to some embodiments of the present invention;

[0023] Figure 5 The body surface contour reconstructed by the optical body surface system;

[0024] Figure 6 The outer contour of the body surface is obtained by delineating the CT image;

[0025] Figure 7 Position information of the patient's heart relative to the body surface during this phase;

[0026] Figure 8 The position information of the patient's left lung relative to the body surface at this phase. DETAILED DESCRIPTION

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present invention. The technical features of the various embodiments can be combined with each other to form a practical solution to achieve the purpose of the invention. For ordinary technicians in this field, the present invention can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0028] It should be understood that the "system" and "unit" used herein are a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions. In addition, the "system" and "unit" can be implemented by software or hardware, and can be a physical or virtual name for the functional part.

[0029] The present invention uses a flow chart to illustrate the operations performed by the system according to an embodiment of the present invention. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes. The technical solutions in the various embodiments may be combined with each other to achieve the purpose of the present invention.

[0030] Embodiment 1:

[0031] A system that uses optical body surface motion signals to synthesize real-time images, such as Figure 1 As shown, it includes: an acquisition unit, which is used to acquire the images of the patient before treatment and the real-time optical surface data during the treatment; a synthesis unit, which is used to synthesize the acquired images of the patient before treatment and the real-time optical surface data during the treatment into a real-time image synchronized with the change of the optical surface motion signal according to a certain mapping relationship.

[0032] Obtain images of the patient before treatment. The closer the images are to the day of treatment, the better the effect. The best images are those taken on the day before treatment.

[0033] Acquiring images of the patient before treatment includes but is not limited to collecting four-dimensional images obtained by four-dimensional cone beam CT (4D-CBCT) or other means before treatment, or reconstructing four-dimensional images before treatment using original kilovolt (kV) two-dimensional projections of three-dimensional cone beam CT (3D-CBCT) or two-dimensional or three-dimensional images obtained by other means and positioning four-dimensional computed tomography (4D-CT) or four-dimensional magnetic resonance imaging (4D-MRI) and other four-dimensional images.

[0034] The optical body surface data is patient body surface data collected by an optical body surface imaging system, etc. One embodiment is to divide the body surface into grids, etc., and obtain coordinate data of each point on the grid, which may include triangular grids, quadrilateral grids, etc., as long as the body surface can be divided to obtain coordinate data.

[0035] One embodiment is to use an optical body surface imaging system to first project LED light onto the patient's body surface, then obtain the reflected light from the patient's body surface through a camera, and generate dynamic 3D body surface information through real-time reflected light. The 3D body surface reconstruction is based on the principle of triangulation and can be used to calculate the patient's real-time six-dimensional body surface movement: translational movement: front and back, left and right, and up and down; rotational movement: around the x-axis, around the y-axis, and around the z-axis.

[0036] An illustrative embodiment is:

[0037] (1) Collect the patient's pre-treatment four-dimensional cone-beam CT (4D-CBCT) data, or use the original kilovolt (kV) two-dimensional projection of three-dimensional cone-beam CT (3D-CBCT) and the positioning four-dimensional computed tomography (4D-CT) or four-dimensional magnetic resonance imaging (4D-MRI) phase data for data processing to reconstruct the pre-treatment 4D image.

[0038] (2) During the treatment process, the optical body surface data at time t is collected in real time and standardized into a 3D body surface mask map or a 2D body surface height map, or other standard data formats.

[0039] (3) Based on the target patient's pre-treatment 4D-CBCT or reconstructed 4D fusion image, and the real-time optical body surface standardized data during treatment, a synthetic real-time dynamic 4D-CBCT image with the change of optical body surface motion signal as the time phase is obtained according to a certain mapping relationship.

[0040] The present invention utilizes the characteristics of in vivo motion images such as 4D images reconstructed from 4D images such as 4D-CBCT, 4D-CT, 4D-MR or 3D images before the patient's treatment and the optical body surface 4D respiratory motion signal to establish a dynamic mapping relationship, and then utilizes the cone beam CT before treatment and the synchronous optical body surface motion signal during treatment to synthesize real-time four-dimensional cone beam CT. Realize real-time, non-invasive, dynamic visualization monitoring of the internal structure of the in vivo motion target area. Based on the universal accelerator platform, a real-time monitoring solution for motion target areas is provided with lower economic cost, wider patient application range, and better compatibility with existing sites and traditional quality control equipment, thereby improving the accuracy and timeliness of radiotherapy of motion target areas and reducing the risk of tumor recurrence and metastasis and radiation damage to normal organs due to off-target.

[0041] Embodiment 2:

[0042] The image obtained before the treatment of the patient is a four-dimensional image obtained directly before the treatment, or a four-dimensional image reconstructed from a three-dimensional image of the patient before the treatment.

[0043] The acquired four-dimensional images before treatment, including but not limited to four-dimensional cone beam CT (4D-CBCT), 4D-CT, 4D-MR, etc., can be directly used to construct pre-treatment multimodal 4D image training set data.

[0044] The more common three-dimensional cone beam CT (3D-CBCT) in clinical practice also contains a lot of useful information such as changes in the patient's anatomical structure. The original kilovolt (kV) two-dimensional projection of three-dimensional cone beam CT (3D-CBCT) and the phase data of four-dimensional images such as positioning four-dimensional computed tomography (4D-CT) or four-dimensional magnetic resonance imaging (4D-MRI) can be used for model training data processing to reconstruct the four-dimensional image before treatment.

[0045] One of the exemplary embodiments of the present invention uses the instantaneous kV two-dimensional projection raw data of 3D-CBCT to associate with the motion signal of 4D-CT. The patient's 4D-CT prior anatomical structure is used to make up for the problem of insufficient number of instantaneous kV two-dimensional projections, while retaining the anatomical information reflected by the kV two-dimensional projection on the day of treatment or the day closest to the treatment. The specific method is: first, a certain phase of 4D-CT is used as a reference (I0), and the new image can be expressed as:

[0046] I(I,j,k)=F(I0,D)=I0(i+D x (I,j,k),j+D y (I,j,k),k+D z (I,j,k))

[0047] Where (I,j,k) is the voxel position, D is the deformation of the new image I relative to I0, and D x , D y , D z are the components of deformation D in the x, y, and z directions respectively. Calculate the deformation between each phase of 4D-CT and I0, and calculate its average deformation And the principal component analysis is performed to obtain the first three principal components D1, D2 and D3 of these deformations, then the deformation D of Ⅰ relative to I0 can be simplified as:

[0048] Where W1, W2 and W3 are the weights of each principal component. To search for the kV two-dimensional projection (P) of a certain phase, the deformation of the 4D image of the phase relative to I0 should satisfy DRR(F(I0,D))=P

[0049] Where DRR represents the digital reconstruction projection of I. The weights of the principal components of each deformation can be obtained by solving the above equation through gradient descent, and the 4D image can be estimated. Subsequently, the B-spline basis can be further used to fine-tune the deformation to make I more consistent with the simultaneous phase kV two-dimensional projection.

[0050] It is also possible to use 4D-MRI to correlate the instantaneous kV two-dimensional projection of 3D-CBCT to obtain 4D images, and 4D-MRI has good soft tissue resolution information, so as to obtain the patient's true four-dimensional anatomical structure during the treatment process.

[0051] Before synthesizing the acquired images of the patient before treatment and the real-time optical body surface data during treatment into a real-time image synchronized with the change of the optical body surface motion signal according to a certain mapping relationship, the real-time optical body surface data is standardized.

[0052] The CT, MR or CBCT images of patients are usually stored in accordance with the DICOM standard to distribute voxels on a three-dimensional rectangular coordinate system, while the optical surface imaging system triangulates the surface images collected by the sensor into triangulated data, and the distribution of triangles varies greatly with the patient's body shape. Although deep learning neural networks do not have strict restrictions on the size, direction, resolution, etc. of input images, they are also required to have a unified basic format. The network trained by the present invention requires surface data as input. In order to facilitate subsequent model training, the surface contour data extracted from the optical surface data and CBCT image data are converted into standardized data for association. The quasi-standardized representation of the surface data is as follows: Figure 3 The two-dimensional height distribution of each point on the body surface relative to the coronal plane where the isocenter is located, or as shown in Figure 4 The three-dimensional mask images of the body surface contours shown are both input data for deep learning neural networks.

[0053] The method to convert DICOM data into standardized data is to write a program based on Python's Pydicom, SimpleITK and Sickit-Image packages and other tools with similar functions for processing medical images, extract the CONTOUR DATA tags in DICOM, obtain the boundary line of the body surface, use the polygon algorithm to obtain the points within the boundary line, and count the number of points on the body surface from the isocenter height in the direction perpendicular to the coronal plane to obtain the two-dimensional height distribution of the body surface, or directly generate a three-dimensional body surface mask image.

[0054] The method for standardizing optical body surface signals is to project the endpoints of the triangle element onto the coronal plane where the isocenter is located, obtain the body surface height at the projection point based on the point coordinates, and then interpolate it to the entire plane to obtain a two-dimensional height distribution map. The body surface boundary can be calibrated on each fault based on the body surface height, and further generate a three-dimensional surface mask map.

[0055] It also includes a deep learning network model unit with the mapping relationship, which is used to obtain images of a certain phase i before treatment of the patient and optical surface data of phase j during treatment (j≠i), input the deep learning network model, output the image of phase j, and combine the images of consecutive phases j to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal.

[0056] The 4D-CBCT images scanned before treatment and the optical body surface data collected in real time during treatment were standardized and used as inputs to the deep learning network model to obtain a synthetic real-time dynamic 4D-CBCT synchronized with the optical body surface motion signal. For patients who only had conventional 3D-CBCT, their kV two-dimensional projection combined with 4D images reconstructed by 4D-CT or 4D-MRI were used as alternative data for 4D-CBCT and input into the deep learning network model to obtain a synthetic real-time dynamic 4D-CBCT synchronized with the optical body surface motion signal.

[0057] The deep learning network model is set at the local treatment end of the system that uses optical body surface motion signals to synthesize real-time images, or at the server end, or in the cloud. The deep learning network model is a trained model, or a network model that continuously self-learns based on continuously updated 4D-CBCT images and optical body surface data.

[0058] The present invention utilizes deep learning technology to mine and analyze the characteristic associations of a large number of patients' 4D-CBCT, 4D-CT, 4D-MR and other in vivo motion images before treatment and the optical body surface 4D respiratory motion signals during treatment, and establishes a dynamic mapping relationship, and then utilizes the cone-beam CT before treatment and the synchronous optical body surface motion signals during treatment to synthesize real-time four-dimensional cone-beam CT images.

[0059] Embodiment three:

[0060] A method for synthesizing real-time images using optical body surface motion signals and a system for synthesizing real-time images using optical body surface motion signals have a one-to-one correspondence, and the embodiments refer to embodiments 1 to 2.

[0061] Embodiment 4:

[0062] A deep learning network model training system for synthesizing real-time images using optical body surface motion signals, including a deep learning network model unit, which is used to train the deep learning network model to obtain the mapping relationship between the patient's body surface and internal structure data; the deep learning network model is trained on a four-dimensional image data set, the deep learning network model input is a four-dimensional image of a certain phase i and optical body surface data or body surface contour data of another phase j, and the deep learning network model output is an image of phase j. Figure 5-8 As shown in the figure, a positioning four-dimensional CT image of a patient at a certain time phase and its corresponding real-time body surface information during treatment obtained by the optical body surface system are shown. Figure 5 is the body surface contour reconstructed by the optical body surface system. Figure 6 The outer contour of the body surface is obtained by delineating the CT image. Figure 7 and Figure 8 The position information of the patient's heart and left lung relative to the body surface at this phase is respectively displayed.

[0063] Deep learning network model training set data preprocessing:

[0064] (1) Mining and analyzing a large number of in vivo motion images such as 4D-CBCT, 4D-CT, and 4D-MR of patients before treatment; or making full use of the large amount of useful data contained in conventional 3D-CBCT and making the model more applicable, the present invention uses the patient's 4D-CT (if available, 4D-MRI can also be included) prior anatomical structure and the anatomical information of the day of treatment reflected by the kV two-dimensional projection collected during CBCT scanning to construct a pre-treatment multimodal 4D fusion image training set.

[0065] 4D-CT, 4D-MR and 4D-CBCT can be directly used to construct multimodal 4D image training set data before treatment. 3D-CBCT, which is more common in clinical practice, also contains a lot of useful information such as changes in the patient's anatomical structure. In order to overcome the averaging effect of the longer imaging time of 3D-CBCT on motion, this embodiment uses the instantaneous kV two-dimensional projection raw data of CBCT to associate with the motion signal of 4D-CT. The patient's 4D-CT prior anatomical structure is used to make up for the problem of insufficient number of instantaneous kV two-dimensional projections, while retaining the anatomical information on the day of treatment reflected by the kV two-dimensional projection. The specific method of reconstructing 3D images into 4D images can adopt the method of Example 2.

[0066] (2) The body surface contour data extracted from the optical body surface data or CBCT image data of different patients are converted into standardized data as the input data of the deep learning model. Optical body surface data is to divide the body surface into grids, etc., and obtain the coordinate data of each point on the grid. Body surface contour data is the outer contour data of the skin on a three-dimensional or four-dimensional image.

[0067] The CT, MR or CBCT images of patients are usually stored in accordance with the DICOM standard to distribute voxels on a three-dimensional rectangular coordinate system, while the optical body surface system triangulates the body surface images collected by the sensor into triangulated data, and the distribution of triangles varies greatly with the patient's body shape. Although deep learning neural networks do not have strict restrictions on the size, direction, resolution, etc. of input images, they are also required to have a unified basic format. The network trained by the present invention requires body surface data as input. In order to facilitate subsequent model training, the body surface contour data extracted from the optical body surface signal and the CBCT image data are converted into standardized data for association. The quasi-standardized representation of the body surface data is as follows: Figure 3 The two-dimensional height distribution of each point on the body surface relative to the coronal plane where the isocenter is located, or as shown in Figure 4 The three-dimensional mask images of the body surface contours shown are both input data for deep learning neural networks.

[0068] The method of converting DICOM data into standardized data is to write a program based on Python's Pydicom, SimpleITK and Sickit-Image packages and other tools with similar functions for processing medical images, extract the CONTOUR DATA tag in DICOM, obtain the boundary line of the body surface, use the polygon algorithm to obtain the points within the boundary line, and count the number of points on the body surface from the height of the isocenter in the direction perpendicular to the coronal plane to obtain the two-dimensional height distribution of the body surface, or directly generate a three-dimensional body surface mask map. The method of standardizing optical body surface signals is to project the endpoints of the triangle element onto the coronal plane where the isocenter is located, obtain the body surface height at the projection point based on the point coordinates, and then interpolate to the entire plane to obtain a two-dimensional height distribution map. The body surface boundary can be calibrated on each fault based on the body surface height, and further generate a three-dimensional body surface mask map.

[0069] Direct reconstruction of the skin 4D optical surface can obtain richer real-time dynamic spatial data. For example, the Catalyst (C-Rad, Sweden) system used in this embodiment can achieve sub-millimeter surface positioning accuracy and long-term stability, and support non-rigid registration with 6 degrees of freedom. Although it is still subject to the accuracy limitations and individual differences of the surface indirect signal itself, combining a large amount of patient 4D surface motion data with other multimodal 4D in-vivo dynamic images through deep learning technology can provide richer and more accurate correlation features for establishing a synchronous dynamic mapping model between the surface and the body.

[0070] Construction and application of deep learning network model: The deep learning model of the 4D fusion image dataset was modified by adding a layer of input, namely, standardized data of surface features. It was retrained on the above 4D fusion image dataset to establish an image generation model to dynamically map the optical surface motion signals with the in vivo anatomical structures contained in the fusion images.

[0071] Model training phase:

[0072] The existing deep learning image generation deep learning network model is modified, an additional layer of input, namely body surface normalization data, is added, and retrained on the 4D fusion image dataset. The input of the deep learning network model is the fusion image of a certain phase (such as phase i) and the 3D body surface mask map or 2D body surface height map of another phase j (j≠i), and the output is the 3D-CBCT map of phase j (mainly to ensure the accuracy of the target area position). The loss function uses the sum of squared deviations with an L1 regularization term, and the model parameters are iteratively optimized according to the gradient of the loss function. The model output is a synthetic real-time dynamic 4D-CBCT with the change of optical body surface motion signal as the phase.

[0073] The deep learning network model used in this embodiment is a gradually progressive three-set model, including a U-type convolutional neural network, a generative adversarial network, and a deep recurrent neural network, making full use of the advantages and characteristics of various models. Among them, the U-type convolutional neural network is a simpler image generation network, which adopts a U-shaped architecture. There are jump links in the corresponding Downsampling and Upsampling layers at both ends, so that the model can make more full use of the multi-scale features of the image, and the output of the model is an image of the same size as the input image. The generative adversarial network (GAN) uses two convolutional neural networks as image generators and classifiers respectively, and improves the prediction effect of the generator through the antagonism of the two. The present invention uses the trained U-type convolutional neural network as the generator of the GAN model, and adds a convolutional neural network as a classifier to improve the model prediction effect. On this basis, the deep recurrent neural network can be further used to mine the association between different time phases to solve the problem of time series signal correlation. In this model, the output at a certain moment will be used as one of the inputs at the next moment or several moments later and affect its output, thereby extracting the self-correlation information of the time series signal. After a more complete image generation network is obtained through the training of the first two models, the body surface data can be input as a periodic signal, and the image will be used as one of the inputs at the next moment. The CBCT image output at each moment will be used as one of the inputs at the next moment.

[0074] Embodiment five:

[0075] A deep learning network model training method for synthesizing real-time images using optical body surface motion signals, such as Figure 5 As shown, there is a one-to-one correspondence with a deep learning network model training system that uses optical body surface motion signals to synthesize real-time images. For specific description, see Example 6.

[0076] The technical solution of the present invention obtains a synthetic real-time dynamic 4D-CBCT with the change of optical body surface motion signal as the time phase, and the result is verified by the following method, which is accurate, real-time and reliable, and can be applied to conventional radiotherapy.

[0077] (1) Using the 4DXCAT digital body mask, we simulated the patient's normal / abnormal breathing pattern anatomical structure and movement changes by setting different parameters in the software, generated virtual patient images corresponding to different imaging modalities (4D-CT, 4D-MRI and 4D-CBCT) and applied them to the model to preliminarily verify the accuracy of the model.

[0078] (2) Synchronously acquire the optical surface signals and 4D-CBCT data of the simulated motion phantom. The 4D-CT and 4D-MRI data of the same phantom are used as model inputs, and the synthetic 4D-CBCT output by the model is compared with the known 4D-CBCT at the same time to evaluate the accuracy of the method. Adjust the respiratory phase setting of the phantom. The simulated motion phantom can also input the real breathing patterns of different patients to better simulate complex clinical scenarios, and the accumulated data can be used as a supplement to the model training set.

[0079] (3) A navigation gold beacon was implanted in the motion plug-in of the simulation phantom. The accuracy and sensitivity of synthetic 4D-CBCT in monitoring the position of the moving target area were evaluated using the Beacon radio frequency wave position tracked by the Calypso real-time positioning system or the real-time gold beacon position obtained by fluoroscopic imaging (simulating the implantation of an ordinary gold beacon without radio frequency wave transmission function) as a reference.

[0080] The above verification results are used as feedback data to further debug and iteratively optimize the model.

[0081] As a large modern tertiary-level Class A cancer specialty hospital integrating clinical, scientific research, teaching and prevention, the applicant, Peking University Cancer Hospital, had an outpatient volume of 730,000 in 2019. While providing patients with first-class cancer diagnosis and treatment services, it has established an advanced multimodal medical database and interdisciplinary research collaboration platform based on the Key Laboratory of Malignant Tumor Pathogenesis and Translational Research of the Ministry of Education, providing this project with sufficient cases and data resources for model training and verification. The preliminary experimental data preliminarily supported the technical solution of the present invention, which has good feasibility as a whole. Based on the data of 500 chest radiotherapy patients who used the optical surface guidance system during the treatment process, the patient's positioning four-dimensional CT and the surface information obtained in real time by the optical surface were used to verify that the surface and the movement of the internal structure have a good mapping relationship, with an accuracy of more than 95%, achieving the expected effect of mapping the changes in the anatomical structure in the body through the surface movement.

[0082] The beneficial effects that may be brought about by the embodiments of the present invention include but are not limited to: the present invention establishes an individualized dynamic mapping relationship between body surface signal characteristics and the position of the in vivo motion target area by fusing multimodal medical big data to train a deep learning network model, and then uses the cone beam CT before treatment and the optical body surface motion signal during treatment to synthesize a real-time dynamic 4D-CBCT synchronized with the optical body surface motion signal, which is used to guide the monitoring of the motion target area in precise radiotherapy, and provide medical evidence and scientific methods for clinical decision-making. Compared with traditional 4D-imaging, the 4D-CBCT synthesized by the present invention using artificial intelligence technology is closer to the real anatomical structure and respiratory movement of the patient during treatment. The synthetic real-time 4D-CBCT can be used as a complementary technology for complex integrated equipment such as MR-accelerators, and can realize accurate, real-time, non-invasive, visual dynamic tracking and no additional dose monitoring of the motion target area during radiotherapy on the existing traditional accelerator platform, with the advantages of low cost, good compatibility and universality. It is conducive to clinical promotion and industrial application.

[0083] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.

[0084] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation of the present invention.

[0085] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences, the use of alphanumeric characters, or the use of other names described in the present invention are not intended to limit the order of the processes and methods of the present invention. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of the present invention.

[0086] Similarly, it should be noted that in order to simplify the description of the present invention and thus facilitate the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes grouped into one embodiment, drawing or description thereof. However, this disclosure method does not mean that the subject matter of the present invention requires more features than those mentioned in the claims.

[0087] Finally, it should be understood that the embodiments described in the present invention are only used to illustrate the principles of the embodiments of the present invention. Other variations may also fall within the scope of the present invention. Therefore, as an example and not a limitation, the alternative configurations of the embodiments of the present invention may be considered consistent with the teachings of the present invention. Accordingly, the embodiments of the present invention are not limited to the embodiments explicitly introduced and described in the present invention.

Claims

1. A system for synthesizing real-time images using optical body surface motion signals, characterized in that: include: An acquisition unit, used to acquire images of the patient before treatment and real-time optical body surface data during treatment; A deep learning network model unit is used to input the images of the patient before treatment and the real-time optical surface data during the treatment process into the deep learning network model, and synthesize them into real-time images synchronized with the changes in the optical surface motion signal. The images of continuous phases are combined to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal; the image of the patient before treatment is a four-dimensional in vivo anatomical structure image directly obtained before treatment, or a four-dimensional in vivo anatomical structure image reconstructed from a three-dimensional image of the patient before treatment; the real-time optical surface data is dynamic three-dimensional surface data generated by obtaining real-time reflected light from the patient's body surface through a camera.

2. The system according to claim 1, characterized in that Before synthesizing the acquired images of the patient before treatment and the real-time optical body surface data during treatment into a real-time image synchronized with the change of the optical body surface motion signal according to a certain mapping relationship, the real-time optical body surface data is standardized.

3. The system according to claim 2, characterized in that It also includes a deep learning network model unit with the mapping relationship, which is used to obtain images of a certain phase i before the patient's treatment and optical surface data of phase j during the treatment process, input the deep learning network model, and output the image of phase j. The images of consecutive phases j are combined to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal.

4. A method for synthesizing real-time images using optical body surface motion signals, characterized in that: Acquire images of the patient before treatment and real-time optical surface data during treatment; input the acquired images of the patient before treatment and real-time optical surface data during treatment into a deep learning network model to synthesize them into real-time images synchronized with changes in optical surface motion signals, and combine images of continuous phases to obtain real-time dynamic four-dimensional images synchronized with changes in optical surface motion signals; the images acquired before treatment of the patient are directly acquired four-dimensional images before treatment, or four-dimensional images reconstructed from three-dimensional images of the patient before treatment; the real-time optical surface data are dynamic three-dimensional surface data generated by acquiring real-time reflected light from the patient's body surface through a camera.

5. The method according to claim 4, characterized in that Before synthesizing the acquired images of the patient before treatment and the real-time optical body surface data during treatment into a real-time image synchronized with the change of the optical body surface motion signal according to a certain mapping relationship, the real-time optical body surface data is standardized.

6. The method according to claim 5, characterized in that The mapping relationship is obtained through a deep learning network model, and the image of a certain phase i before the patient's treatment and the optical surface data of phase j during the treatment are obtained. The deep learning network model is input, and the image of phase j is output. The images of consecutive phases j are combined to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal.

7. A deep learning network model training system for synthesizing real-time images using optical body surface motion signals, characterized in that: The system comprises a deep learning network model unit, which is used to input the acquired images and optical body surface data of the patient before treatment into the deep learning network model, synthesize them into real-time images synchronized with the changes of the optical body surface motion signal, and combine the images of continuous phases to obtain a real-time dynamic four-dimensional image synchronized with the changes of the optical body surface motion signal; the deep learning network model is trained on a four-dimensional image data set, the input of the deep learning network model is a four-dimensional image of a certain phase i and the optical body surface data or body surface outer contour data of another phase j, and the output of the deep learning network model is an image of phase j; the acquired images of the patient before treatment are four-dimensional in vivo anatomical structure images directly acquired before treatment, or four-dimensional in vivo anatomical structure images reconstructed from three-dimensional images of the patient before treatment; the optical body surface data are dynamic three-dimensional body surface data generated by obtaining reflected light from the patient's body surface through a camera.

8. A deep learning network model training method for synthesizing real-time images using optical body surface motion signals, characterized in that: A deep learning network model is trained, and the images and optical surface data of the patient obtained before treatment are input into the deep learning network model to be synthesized into a real-time image synchronized with the changes in the optical surface motion signal. The images of continuous phases are combined to obtain a real-time dynamic four-dimensional image synchronized with the changes in the optical surface motion signal; the deep learning network model is trained on a four-dimensional image data set, and the input of the deep learning network model is a four-dimensional image of a certain phase i and optical surface data or surface contour data of another phase j, and the output of the deep learning network model is an image of phase j; the image of the patient obtained before treatment is a four-dimensional in vivo anatomical structure image directly obtained before treatment, or a four-dimensional in vivo anatomical structure image reconstructed from a three-dimensional image of the patient obtained before treatment; the optical surface data is dynamic three-dimensional surface data generated by obtaining reflected light from the patient's body surface through a camera.

Images