Radiotherapy positioning system and method based on mixed reality

Abstract

The invention relates to the technical field of radiotherapy positioning, in particular to a radiotherapy positioning system and method based on mixed reality. The radiotherapy positioning system comprises a positioning CT three-dimensional model reconstruction module, a point cloud model reconstruction module, a point cloud model feature extraction module and a mixed reality module, and the mixed reality module comprises an environment construction module, a point cloud model matching module, a positioning CT three-dimensional model positioning module and a model error calculation module. According to the radiotherapy positioning method, a position error between a CT three-dimensional model and a point cloud model is calculated and positioned during radiotherapy positioning, positioning error data is obtained, and a patient is guided to position according to the positioning error data. The method has the advantages that errors caused by human factors can be avoided, and the method is more visual. And the point cloud model fused with the patient is used for replacing a real patient for radiotherapy positioning, so that the influence of organ movement in a single treatment process and patient body position deviation between fractional radiotherapy on the radiotherapy curative effect can be avoided as far as possible.

Description

Technical Field

[0001] The present invention relates to the field of radiotherapy positioning technology, and in particular to a radiotherapy positioning system and method based on mixed reality. Background Art

[0002] Accurate and reproducible body positioning during radiotherapy is crucial for improving tumor control and reducing complications in normal tissues. Typically, patient positioning is achieved by aligning marked lines on the patient's body with the laser beam of the accelerator. However, during a single radiotherapy treatment, organ movement caused by functional movements such as breathing, as well as patient positional deviations between radiotherapy fractions, can cause varying degrees of deformation in the tumor and surrounding normal tissues and organs, leading to inaccurate doses to the tumor target or increased doses to normal tissues and organs. Furthermore, during the radiotherapy cycle, the patient's body shape and tumor morphology change, leading to certain deviations. Currently, this can only be addressed in phases through repeated imaging scans to obtain updated data. Mixed reality technology, due to its advantages in 3D visualization, is also increasingly being used in radiotherapy. However, current mixed reality-based radiotherapy positioning technologies rely on artificial vision, which is prone to visual errors.

[0003] Therefore, there is an urgent need for an improved radiotherapy positioning system based on mixed reality to resolve errors caused by human factors. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that radiotherapy positioning technologies based on mixed reality are all based on artificial vision for radiotherapy positioning, which is prone to visual errors.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a radiotherapy positioning system based on mixed reality, comprising:

[0006] A positioning CT three-dimensional model reconstruction module is used to reconstruct a three-dimensional model using the patient positioning CT image. The three-dimensional model is a positioning CT three-dimensional model.

[0007] Point cloud model reconstruction module, the point cloud model reconstruction module collects the patient's body surface point cloud through the camera and reconstructs a three-dimensional model through the point cloud. The three-dimensional model is a point cloud model;

[0008] Point cloud model feature extraction module, which is used to extract surface features of the point cloud model;

[0009] Mixed reality module, which includes environment construction module, point cloud model matching module, CT 3D model positioning module and model error calculation module;

[0010] The environment construction module is used to build and update the 3D map of the environment so that the positioning CT 3D model and point cloud model can be superimposed in the user's field of view;

[0011] The point cloud model matching module is used to extract the surface features of the patient in the user's field of view, and then perform feature matching between the extracted surface features of the patient and the surface features of the point cloud model, so that the point cloud model and the patient in the user's field of view are integrated;

[0012] The positioning CT three-dimensional model positioning module is used to identify the accelerator treatment isocenter and position the positioning CT three-dimensional model at the accelerator treatment isocenter in the user's field of view;

[0013] The model error calculation module is used to calculate the position error of the point cloud model fused with the patient and the positioning CT three-dimensional model positioned at the accelerator treatment isocenter to obtain positioning error data.

[0014] In some embodiments, optionally, the mixed reality-based radiotherapy positioning system further includes an automated positioning guidance module, which is used to connect to the accelerator treatment bed and automatically guide the movement of the accelerator treatment bed when the positioning error data exceeds a threshold value, so that the positioning error data is within the threshold value.

[0015] In some embodiments, optionally, the point cloud model feature extraction module extracts surface features by the following method: first, target recognition is performed, where the target is a point cloud model; then, feature extraction of the target is performed using the ORB algorithm, where the extracted features are ORB features; then, the ORB features are processed using a pre-trained DenseNet model to extract the depth features of the ORB features; finally, the ORB features and the depth features of the ORB features are fused to obtain surface features for feature matching.

[0016] In some embodiments, optionally, the point cloud model matching module extracts surface features by the following method: first, target identification is performed, where the target is the corresponding area on the patient's body surface, and then features of the target are extracted using the ORB algorithm to obtain surface features for feature matching.

[0017] In some embodiments, optionally, when performing target recognition, a Vuforia algorithm is used for target recognition.

[0018] In some embodiments, optionally, before the point cloud model matching module performs target recognition, the recognition angle is calibrated according to preset recognition angles based on different areas of the patient. After the recognition angle is calibrated, target recognition and subsequent feature extraction and feature matching are performed.

[0019] A radiotherapy positioning method based on mixed reality reconstructs a point cloud model of the patient's body surface and a positioning CT three-dimensional model of the patient before radiotherapy positioning. Then, during radiotherapy positioning, the positioning CT three-dimensional model is positioned at the accelerator treatment isocenter, and the point cloud model is fused with the patient by matching surface features. The position error between the positioning CT three-dimensional model and the point cloud model is then calculated to obtain positioning error data, and the patient positioning is guided according to the positioning error data.

[0020] The beneficial effect of the present invention is that radiotherapy positioning is performed by using six-dimensional error data between the point cloud model fused with the patient and the positioning CT three-dimensional model, which can avoid errors caused by human factors and is more intuitive.

[0021] Moreover, by using a point cloud model fused with the patient instead of the real patient for radiotherapy positioning, the effects of organ movement during a single treatment and patient position deviation between radiotherapy fractions on the efficacy of radiotherapy can be avoided as much as possible. For example, for breast patients, the target position movement caused by inconsistencies in limbs and soft tissues can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention will be further described below with reference to the accompanying drawings and examples;

[0023] Figure 1 This is a flow chart of the radiotherapy positioning method based on mixed reality of the present invention. DETAILED DESCRIPTION

[0024] like Figure 1 As shown, a mixed reality-based radiotherapy positioning method reconstructs a point cloud model of the patient's body surface and a 3D positioning CT model of the patient before radiotherapy positioning. During radiotherapy positioning, the 3D positioning CT model is positioned at the accelerator treatment isocenter. The point cloud model and the patient are fused by matching surface features. The position error between the 3D positioning CT model and the point cloud model is then calculated to obtain positioning error data, which is then used to guide patient positioning. The positioning error data is specifically six-dimensional error data. If the six-dimensional error data is not within a threshold, positioning is guided based on the six-dimensional error data until the six-dimensional error data enters the threshold before radiotherapy begins.

[0025] The mixed reality-based radiotherapy positioning system used to implement this method includes a positioning CT three-dimensional model reconstruction module, a point cloud model reconstruction module, a point cloud model feature extraction module and a mixed reality module. The mixed reality module includes an environment construction module, a point cloud model matching module, a positioning CT three-dimensional model positioning module and a model error calculation module.

[0026] Among them, the positioning CT three-dimensional model reconstruction module is used to reconstruct a three-dimensional model through the patient positioning CT image, and the three-dimensional model is a positioning CT three-dimensional model.

[0027] The point cloud model reconstruction module collects the patient's body surface point cloud through a camera and reconstructs a three-dimensional model from the point cloud. The three-dimensional model is a point cloud model.

[0028] The point cloud model feature extraction module is used to extract the surface features of the point cloud model.

[0029] The environment construction module is used to build and update the three-dimensional map of the environment so that the positioning CT three-dimensional model and point cloud model can be superimposed in the user's field of view.

[0030] The point cloud model matching module is used to extract the surface features of the patient in the user's field of view, and then perform feature matching on the extracted surface features of the patient with the surface features of the point cloud model, so that the point cloud model and the patient in the user's field of view are fused together.

[0031] The positioning CT three-dimensional model positioning module is used to identify the accelerator treatment isocenter and position the positioning CT three-dimensional model at the accelerator treatment isocenter in the user's field of view.

[0032] The model error calculation module is used to calculate the position error of the point cloud model fused with the patient and the positioning CT three-dimensional model positioned at the accelerator treatment isocenter to obtain positioning error data.

[0033] The mixed reality module in this embodiment is specifically a HoloLens head-mounted display device. The functions of the positioning CT 3D model reconstruction module, the point cloud model reconstruction module, and the point cloud model feature extraction module can be implemented by computer software. The functions of the environment construction module, the point cloud model matching module, the positioning CT 3D model positioning module, and the model error calculation module can be implemented by software in the HoloLens head-mounted display device.

[0034] The mixed reality-based radiotherapy positioning system of this embodiment also includes an automated positioning guidance module, which is used to connect to the accelerator treatment bed and automatically guide the movement of the accelerator treatment bed when the positioning error data exceeds a threshold value, so that the positioning error data is within the threshold value.

[0035] The positioning error data obtained by the model error calculation module can be superimposed on the user's field of view to achieve more intuitive positioning guidance.

[0036] The point cloud model feature extraction module extracts surface features using the following method: first, target recognition is performed, where the target is the point cloud model. Then, the ORB algorithm is used to extract features of the target, and the extracted features are ORB features. Then, the pre-trained DenseNet model is used to process the ORB features and extract the deep features of the ORB features. Finally, the ORB features and the deep features of the ORB features are fused to obtain the surface features used for feature matching.

[0037] The point cloud model matching module extracts surface features through the following method: first, target recognition is performed, and the target is the corresponding area on the patient's body surface. Then, the feature extraction of the target is performed through the ORB algorithm to obtain the surface features for feature matching.

[0038] When the point cloud model matching module and the point cloud model matching module perform target recognition, the Vuforia algorithm is used for target recognition.

[0039] Before the point cloud model matching module performs target recognition, the recognition angle is calibrated according to the preset recognition angle based on different areas of the patient (such as the pelvis and chest). After the recognition angle is calibrated, target recognition and subsequent feature extraction and feature matching are performed.

[0040] The mixed reality-based radiotherapy positioning system of the present invention also includes a tablet and a computer that cooperate with the HoloLens head display device to perform virtual screen projection. The computer that performs virtual screen projection is located in the control room of the radiotherapy equipment. The content of the virtual screen projection is the user field of view that can be seen by the user in the HoloLens head display device. The user field of view includes the virtual UI panel, virtual projection and the real environment. Doctors and technicians can monitor the patient's positioning process in the control room, avoiding the need for doctors and technicians to walk back and forth between the control room and the machine room, thereby improving the efficiency of radiotherapy.

[0041] The specific steps for performing radiotherapy swing using the mixed reality-based radiotherapy positioning system are as follows:

[0042] Step 1: Use the positioning CT three-dimensional model reconstruction module to reconstruct the patient's positioning CT three-dimensional model.

[0043] Specifically, step 1 includes: performing a CT scan on the patient from the horizontal, coronal, and sagittal planes, with a scanning voltage of 120 kV, an effective tube current of 20 mA, and a reconstruction layer thickness of 5 mm to obtain a CT image; uploading the CT image to the radiotherapy planning system, outlining the target area, clinical target area, and surrounding important organs and tissues, and exporting it as a DICOM-RT dataset. The outlined DICOM-RT dataset is the positioning CT image, and the positioning CT image is imported into the positioning CT three-dimensional model reconstruction module for three-dimensional reconstruction using medical image processing software to reconstruct a three-dimensional model. This three-dimensional model is the positioning CT three-dimensional model, in which the outer contour of the body surface, the planned target area, the clinical target area, and other surrounding organs and tissues are displayed with appropriate transparency. To ensure that the reconstructed three-dimensional model conforms to the conventional three-dimensional software import format and size conversion, the reconstructed three-dimensional model also needs to be resized and formatted.

[0044] Step 2: Use the point cloud model reconstruction module to reconstruct the point cloud model of the patient's body surface.

[0045] Specifically, step 2 includes: a point cloud model reconstruction module uses multiple depth cameras to collect point clouds of the patient's body surface, and the collected point clouds are aligned and spliced ​​to reconstruct a point cloud model of the patient's body surface.

[0046] In step 2, the point cloud model reconstruction module collects point clouds using three depth cameras mounted on parallel tracks. The three depth cameras are 50 cm apart and secured with universal ball joints. The center depth camera is angled 30° horizontally; the left and right depth cameras are angled 30° vertically and 30° horizontally.

[0047] In step 2, each depth camera consists of a central RGB camera unit for capturing color images and two infrared camera units on either side for capturing depth images. Because some scenes have fewer feature points, the point cloud model reconstruction module also uses an infrared dot matrix projector to emit an infrared dot matrix pattern to increase feature points and enhance imaging quality. The left and right infrared camera units feed image data into the depth camera's built-in depth processor, which uses triangulation based on the principle of binocular ranging to calculate the depth value of each pixel.

[0048] The depth camera obtains the internal and external parameters of the camera through camera calibration, aligns the color image and the depth image, and obtains the point cloud.

[0049] The camera calibration process of the depth camera includes four coordinate system conversions: the world coordinate system is converted to the camera coordinate system through the extrinsic parameter matrix; the camera coordinate system is converted to the pixel coordinate system through the intrinsic parameter matrix; the camera coordinate system is converted to the image coordinate system through the focal length diagonal matrix and the distortion coefficient. The conversion formula is shown in Equation 1.

[0050] The depth camera obtains the point cloud by aligning the color image and the depth image. The depth camera reads the color information and depth information of each pixel at the same time, outputs the corresponding color image and depth image, and uses Equation 2 to calculate the 3D point cloud coordinates of the pixel to obtain the point cloud.

[0051]

[0052] The point cloud model reconstruction module can collect three point clouds from different angles through three depth cameras. The three point clouds are aligned and spliced ​​to reconstruct a point cloud model of the patient's body surface. The specific process is as follows: the three point clouds are named P1, P2, and P3, and there is a certain overlap between any two adjacent point clouds. The best transformation matrix is ​​found between each coherent and overlapping point cloud, and these transformations are accumulated to all point clouds; the coordinate system of the point cloud where P1 is located is set as the world coordinate system, and the point clouds P2, P3 and P1 are aligned respectively. With P1 as the target point cloud, P2 and P3 are aligned to the coordinate system where P1 is located, and the registered transformation matrix is ​​T 12 、T 13 According to Formula 3, any point cloud P k Transform to the coordinate system where P1 is located and get P′ k , point clouds P1, P′2 and P′3 are spliced ​​together to form a complete three-dimensional point cloud model of the patient's body surface.

[0053] T=[R|t] (Equation 3)

[0054] P′2=T 12 P2, P′3=T 13 P3

[0055] Among them, R is the rotation matrix, t is the translation matrix, and T is the transformation matrix obtained by registration.

[0056] Step 3: Use the point cloud model feature extraction module to extract the surface features of the point cloud model.

[0057] Specifically, step 3 is as follows: the point cloud model feature extraction module first performs target recognition, where the target is the point cloud model, and then extracts features of the target using the ORB algorithm, where the extracted features are ORB features; then the ORB features are processed using the pre-trained DenseNet model to extract the deep features of the ORB features; finally, the ORB features and the deep features of the ORB features are fused to obtain surface features for feature matching.

[0058] Combining the deep features extracted by DenseNet with ORB features can further improve the robustness and accuracy of subsequent feature matching steps.

[0059] Step 4: Use the environment construction module to build and update the 3D map of the environment so that the positioning CT 3D model and the point cloud model can be superimposed in the user's field of view.

[0060] Specifically, step 4 includes: realizing self-positioning of the HoloLens head-mounted display device through motion estimation of the camera of the HoloLens head-mounted display device, building and updating a three-dimensional map of the environment, and enabling the positioning CT three-dimensional model and the point cloud model to be superimposed in the user's field of view.

[0061] Step 5: Use the positioning CT three-dimensional model positioning module to position the positioning CT three-dimensional model at the accelerator treatment isocenter in the user's field of view.

[0062] Step 5 specifically includes: using the worn HoloLens head-mounted display device to identify the marker code placed at the accelerator treatment isocenter, obtaining the conversion relationship between the coordinate system in the HoloLens holographic space and the coordinate system in the real space, and accurately displaying the positioning CT three-dimensional model in the HoloLens holographic space at the accelerator treatment isocenter.

[0063] Step 6: Use the point cloud model matching module to fuse the point cloud model with the patient in the user's field of view.

[0064] Step 6 specifically includes: extracting surface features of the patient in the user's field of view through the worn HoloLens head display device, and then matching the extracted surface features of the patient with the surface features of the point cloud model to fuse the point cloud model and the patient together.

[0065] Before the point cloud model matching module performs target recognition, the recognition angle is calibrated according to the preset recognition angle based on different areas of the patient (such as the pelvis and chest). After the recognition angle is calibrated, target recognition and subsequent feature extraction and feature matching are performed.

[0066] Calibrating and setting the recognition angle can improve the effect and speed of feature extraction and feature matching, and avoid recognition errors and unstable recognition problems.

[0067] Step 7: Use the model error calculation module to calculate the position error between the point cloud model fused with the patient and the positioning CT three-dimensional model positioned at the accelerator treatment isocenter. The position error is specifically a six-dimensional error.

[0068] The six-dimensional error calculation method is as follows: using the positive direction of the positioning CT 3D model as a vector with unit 1, and normalizing the positions of the point cloud model and the positioning CT 3D model (normalization is a vector with unit 1 and unchanged direction). Using the dot product and cross product formulas, the front-back, left-right, and top-bottom positions of the point cloud model relative to the positioning CT 3D model are calculated, resulting in the six-dimensional error data of the point cloud model relative to the positioning CT 3D model. The pitch angle is calculated based on the Y axis, the roll angle is calculated based on the X axis, and the yaw angle is calculated based on the Z axis.

[0069] Step 8: Use the automated positioning guidance module to automatically guide positioning.

[0070] Specifically, step 8 includes: if the six-dimensional error data is not within the threshold, automatically guiding the accelerator treatment bed to move according to the six-dimensional error data so that the six-dimensional error data enters the threshold, and then starting radiotherapy.

[0071] In the above steps, step 1 and step 2 can be performed simultaneously, and step 5 and step 6 can be performed simultaneously.

Claims

1. A radiotherapy positioning system based on mixed reality, characterized by: include: A positioning CT three-dimensional model reconstruction module is used to reconstruct a three-dimensional model using the patient's positioning CT image, and the three-dimensional model is a positioning CT three-dimensional model; A point cloud model reconstruction module, which collects a point cloud of the patient's body surface through a camera and reconstructs a three-dimensional model from the point cloud, wherein the three-dimensional model is a point cloud model; A point cloud model feature extraction module, wherein the point cloud model feature extraction module is used to extract surface features of the point cloud model; A mixed reality module, comprising an environment construction module, a point cloud model matching module, a CT three-dimensional model positioning module, and a model error calculation module; The environment construction module is used to construct and update a three-dimensional map of the environment so that the positioning CT three-dimensional model and the point cloud model can be superimposed in the user's field of view; The point cloud model matching module is used to extract surface features of the patient in the user's field of view, and then perform feature matching on the extracted surface features of the patient with the surface features of the point cloud model, so that the point cloud model and the patient in the user's field of view are fused together; The positioning CT three-dimensional model positioning module is used to identify the accelerator treatment isocenter and position the positioning CT three-dimensional model at the accelerator treatment isocenter in the user's field of view; The model error calculation module is used to calculate the position error of the point cloud model fused with the patient and the positioning CT three-dimensional model positioned at the accelerator treatment isocenter to obtain positioning error data.

2. The mixed reality-based radiotherapy positioning system according to claim 1, characterized in that: It also includes an automatic positioning guidance module, which is used to connect to the accelerator treatment bed and automatically guide the accelerator treatment bed to move when the positioning error data exceeds a threshold value, so that the positioning error data is within the threshold value.

3. The mixed reality-based radiotherapy positioning system according to claim 1, characterized in that: The point cloud model feature extraction module extracts surface features by the following method: first, target recognition is performed, where the target is a point cloud model; then, feature extraction of the target is performed using the ORB algorithm, and the extracted features are ORB features; then, the ORB features are processed using a pre-trained DenseNet model to extract the deep features of the ORB features; finally, the ORB features and the deep features of the ORB features are fused to obtain surface features for feature matching.

4. The mixed reality-based radiotherapy positioning system according to claim 1, characterized in that: The point cloud model matching module extracts surface features by the following method: first, target recognition is performed, the target is the corresponding area on the patient's body surface, and then the target is extracted using the ORB algorithm to obtain surface features for feature matching.

5. The mixed reality-based radiotherapy positioning system according to claim 3 or 4, characterized in that: When performing target recognition, the Vuforia algorithm is used for target recognition.

6. The mixed reality-based radiotherapy positioning system according to claim 4, characterized in that: Before the point cloud model matching module performs target recognition, the recognition angle is calibrated according to the preset recognition angle based on different areas of the patient. After the recognition angle is calibrated, target recognition and subsequent feature extraction and feature matching are performed.

7. A radiotherapy positioning method based on mixed reality, characterized by: Before radiotherapy positioning, the point cloud model of the patient's body surface and the patient's positioning CT three-dimensional model are reconstructed separately. Then, during radiotherapy positioning, the positioning CT three-dimensional model is positioned at the accelerator treatment isocenter, and the point cloud model is fused with the patient through surface feature matching. The position error between the positioning CT three-dimensional model and the point cloud model is then calculated to obtain the positioning error data, and the patient positioning is guided based on the positioning error data.

Images