Body surface tracking device in radiotherapy, and general target tracking system and method based on deep learning

Abstract

The invention relates to the field of motion target areas in radiotherapy, and discloses a body surface tracking device in radiotherapy and a general target tracking system and method based on deep learning, the body surface tracking device is used for collecting body surface motion images in real time, and the view field position and range of the body surface tracking device can be adjusted; the image processing unit receives synchronous image information collected by the two visual cameras of the body surface tracking device in the field-of-view detection range in real time, inputs the synchronous image information into the deep learning tracking module, synchronously outputs pixel coordinates of a target area on images of the two visual cameras, and outputs the pixel coordinates of the target area through the coordinate conversion module. Pixel coordinates are converted into three-dimensional space coordinates, relevant information is transmitted to a real-time display module for visual display, and meanwhile, the image or coordinate information is sent to external equipment through an external equipment interface; and the real-time display unit visually displays the image information, the deep learning tracking area, the real-time pixel coordinates and the three-dimensional space coordinates.

Description

Technical Field

[0001] The present invention relates to the field of moving target areas in radiotherapy, and in particular to a body surface tracking device in radiotherapy, and a general target tracking system and method based on deep learning. Background Art

[0002] In recent years, methods for image guidance and dynamic target tracking have emerged one after another. People have developed many different image acquisition methods for image guidance to further ensure the precise delivery of radiation doses, including kV-X-ray imaging, MV-X-ray imaging, magnetic resonance imaging (MRI), cone beam computed tomography (CBCT), ultrasound imaging (US), and implanted radiofrequency generators in the body.

[0003] At present, the most commonly used methods for dynamic target real-time image guidance in clinical practice are kV-X-ray image guidance, implanted radiofrequency generators in the body, and real-time MRI guidance. Since the soft tissue contrast of kV-X-ray imaging is low, it is difficult to distinguish the target area from normal tissue in some cases. Therefore, it is usually clinically adopted to implant metal markers (gold labels) near the target area and detect the position of the gold labels to indirectly reflect the motion information of the target area. However, this method has problems such as image artifacts, poor signal accuracy, the relative position of the gold label and the target area will change over time, and there are trauma. Implanting a radiofrequency generator in the body is a traumatic target tracking method, and the relative relationship between the position of the radiofrequency generator and the position of the target area will also change over time. Real-time MRI guidance is an emerging image guidance method with the characteristics of high soft tissue contrast and the advantages of no radiation and losslessness. However, due to the existence of its magnetic field, it will cause mutual interference with radiotherapy equipment, affecting the imaging quality and radiotherapy dose distribution. Currently, only a few photon radiotherapy equipment have integrated MRI guidance function, and it has not yet been realized on ion radiotherapy equipment (protons, heavy ions, etc.). Surface tracking is an alternative to internal target tracking, including surface optical imaging, optical marker detection, pressure sensors, airflow and temperature change detection, etc. It is currently widely used in tracking the motion of target areas in radiotherapy. Its advantages are non-radiation and non-destructive, high detection frequency, and its disadvantages are that it cannot directly reflect the motion information of the target area, the equipment is expensive, surface markers or auxiliary equipment (pressure sensors, flow detectors, temperature detectors, etc.) are required, and the detection accuracy is low. The combined detection method combines the advantages of direct in vivo detection and indirect surface detection methods. It mainly obtains the internal and surface motion signals at the same time in a short period of time before treatment and establishes a relationship between the two. During the treatment process, the surface signals are detected in real time to calculate the motion position of the target area in the body, and the relationship is updated at regular intervals to ensure the prediction accuracy of the target area position. It can be seen that its origin is to obtain accurate motion signals of the surface and internal target areas.

[0004] Therefore, how to achieve accurate, efficient, non-destructive, label-free, convenient, low-cost real-time detection of the three-dimensional spatial position of target areas on the body surface and inside the body is a key issue in precision radiotherapy of moving targets. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a body surface tracking device in radiotherapy, a general target tracking system and method based on deep learning, which can realize low-cost, high-precision, efficient, intelligent and convenient real-time detection and tracking of moving target areas.

[0006] To achieve the above-mentioned purpose, in the first aspect, the technical solution adopted by the present invention is: a body surface tracking device in radiotherapy, which includes: a shell; a first imaging device and a second imaging device, which are symmetrically arranged back to back and located in the middle of the shell, and the bottom of the two are respectively provided with a first moving component and a second moving component; the first moving component and the second moving component are respectively arranged at the bottom of the shell, for driving the corresponding first imaging device and the second imaging device to move; the first optical reflector and the second optical reflector are respectively arranged at the two ends of the shell through the third moving component and the fourth moving component, and the third moving component and the fourth moving component are respectively arranged on the two end side walls of the shell, the inclination angle of the first optical reflector is adjusted by the third moving component, and the inclination angle of the second optical reflector is adjusted by the fourth moving component; located directly below the first optical reflector and the second optical reflector, a first transmission window and a second transmission window are respectively opened at the bottom of the two ends of the shell to transmit external light signals to the inside of the shell.

[0007] Furthermore, both the first imaging device and the second imaging device use visual cameras to collect body surface image information from two angles in real time.

[0008] Further, a first group of camera indicator lights connected to the first imaging device and a second group of camera indicator lights connected to the second imaging device are provided in the middle portion of the housing; The middle of the shell is provided with a first signal transmission interface and a second signal transmission interface; the middle of the shell is also provided with an external device fixing interface, and the side walls at both ends of the shell are respectively provided with shell fixing interfaces.

[0009] In the second aspect, the technical solution adopted by the present invention is: a general target tracking system based on deep learning in radiotherapy, comprising: a body surface tracking device in radiotherapy as mentioned above, used to collect body surface motion images in real time, and its field of view position and range are adjustable; an image processing unit, which receives in real time the synchronous image information collected by the two sets of visual cameras of the body surface tracking device in the field of view detection range, and inputs it into the deep learning tracking module, synchronously outputs the pixel coordinates of the target target area on the two sets of visual camera images respectively, converts the pixel coordinates into three-dimensional space coordinates through the coordinate transformation module, and transmits the relevant information to the real-time display module for visual display, and sends the image or coordinate information to the external device through the external device interface; a real-time display unit, connected to the image processing unit, visually displays the image information, deep learning tracking area, real-time pixel coordinates and three-dimensional space coordinates transmitted by the real-time display module in the image processing unit.

[0010] Further, in the deep learning tracking module of the image processing unit, the deep learning tracking process includes: A body surface tracking model is selected and trained, wherein the body surface tracking model is a neural network model with a self-attention mechanism; A tracking area is selected on the real-time acquired image, and a trained body surface tracking model is input. The body surface tracking model tracks the selected area in real time to output the real-time pixel coordinate position of the outer frame or outline of the tracking area.

[0011] Furthermore, the body surface tracking model adopts the Transformer model and uses a large data set to train the model to track body surface movements in any scenario.

[0012] Further, the pixel coordinates are converted to three-dimensional space coordinates, including: Two sets of visual cameras are used to simultaneously capture images of the coordinate calibration plate at multiple angles and positions. The internal and external parameter matrices of each camera are obtained through the spatial camera calibration method. The parameter matrix is ​​used to convert the real-time pixel coordinates of the external frame or outline of the tracking area into three-dimensional space coordinates and output.

[0013] In the third aspect, the technical solution adopted by the present invention is: a marker-free in vivo target tracking method, which is implemented based on the above-mentioned general target tracking system based on deep learning in radiotherapy, and includes: using a transfer learning method to fine-tune and optimize the surface tracking model, and its learning data uses in vivo image data; amplifying the in vivo image data, and performing transfer training on the surface tracking model with the amplified data to obtain a general tracking model that can realize surface and in vivo target target tracking; selecting a tracking area on the in vivo image collected in real time by an external device as the target target to be tracked by the general tracking model, and inputting it into the general tracking model; the general tracking model tracks the selected target target in real time, and outputs the real-time pixel coordinate position of the target target, converts the pixel coordinates into three-dimensional space coordinates, and finally obtains the real-time coordinate position of the target target in three-dimensional space, and visualizes the relevant information, and sends the image or coordinate information to the external device through the external device interface.

[0014] Furthermore, the in vivo image data is augmented by using a deformation data augmentation method.

[0015] Furthermore, the deformation data augmentation method includes: According to the deformation field between the image data of adjacent motion phases, a new deformation field between adjacent motion phases is obtained by weighted summation method; The new deformation field is applied to the reference phase image data to generate virtual images between adjacent motion phases, and then rotation enhancement is performed.

[0016] The present invention adopts the above technical solution, which has the following advantages: 1. The body surface tracking device of the present invention utilizes an ordinary visual camera, adopts a dual-camera back-to-back design, and combines an optical reflector to realize binocular visual image acquisition in any area in the space, while achieving the purpose of a compact device structure.

[0017] 2. The target detection and tracking method of the present invention is based on deep learning. The model is trained through big data to obtain a universal tracking model, which enables it to have the ability to track body surface movements in any scenario. At the same time, there is no need to place tracking targets specifically on the body surface. Only the tracking area needs to be selected to achieve accurate, efficient, convenient and low-cost real-time tracking of the target area.

[0018] 3. The present invention obtains a body surface tracking model based on big data training, and uses in vivo imaging data to perform migration training on the model to obtain a universal tracking model for the target area on the body surface and in the body. There is no need to implant markers in the body. Only the area to be tracked needs to be selected to achieve accurate, efficient, non-destructive and convenient real-time tracking of the target area.

[0019] 4. The present invention changes the problems of expensive, poorly accurate, low-efficiency and complex operation of motion detection equipment in traditional radiotherapy, and provides strong support for precision radiotherapy.

[0020] In summary, the present invention can also be applied to other fields, such as surgical navigation, motion monitoring of equipment in treatment rooms, and motion monitoring in other scenarios, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic structural diagram of a body surface tracking device in an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a general target tracking system in an embodiment of the present invention; Figure 3 It is a deep learning tracking process and a pixel coordinate to three-dimensional space coordinate conversion process in the image processing unit of the embodiment of the present invention; Figure 4 is a flow chart of a label-free in vivo target tracking method according to an embodiment of the present invention; Figure 5 This is a diagram of a deep learning network structure for real-time tracking of target areas based on Transformer in an embodiment of the present invention; Figure 6a It is the three-dimensional motion trajectory (tracking) and the reference motion trajectory (NDI) of the body surface in the x direction obtained based on deep learning tracking in the motion state in the embodiment of the present invention; Figure 6b It is the three-dimensional motion trajectory (tracking) and the reference motion trajectory (NDI) of the body surface in the y direction obtained based on deep learning tracking in the motion state in the embodiment of the present invention; Figure 6c It is the three-dimensional motion trajectory (tracking) and the reference motion trajectory (NDI) of the body surface in the z direction obtained based on deep learning tracking in the motion state in the embodiment of the present invention; Figure 7a It is a statistical graph of body surface three-dimensional motion errors in the x direction obtained based on deep learning tracking under 20 motion states tested in an embodiment of the present invention; Figure 7b It is a statistical graph of body surface three-dimensional motion errors in the y direction obtained based on deep learning tracking under 20 motion states tested in an embodiment of the present invention; Figure 7c It is a statistical graph of body surface three-dimensional motion errors in the z direction obtained based on deep learning tracking under 20 motion states tested in an embodiment of the present invention; Figure 8 is a flow chart of a method for deformable amplification of in vivo image data according to an embodiment of the present invention; Figure 9aThis is the tracking situation of a larger target area in an embodiment of the present invention, and the in vivo target area tracking situation tested, wherein the black solid line frame is the target area reference boundary frame, and the red solid line frame is the model tracking frame; Figure 9b This is the tracking situation of a smaller target area in an embodiment of the present invention, and the in vivo target area tracking situation tested, wherein the green solid line frame is the target area reference boundary frame, and the blue solid line frame is the model tracking frame; Fig.10 This is a statistical diagram of target tracking errors in 20 cases tested in an embodiment of the present invention; Reference numerals: 100-body surface tracking device, 101-housing, 102-first imaging device, 103-second imaging device, 104-first motion component (motion component of visual camera), 105-second motion component (motion component of visual camera), 106-first optical reflector, 107-second optical reflector, 108-third motion component (motion component of first optical reflector), 109-fourth motion component (motion component of second optical reflector), 110-first transmission window, 111-second transmission window, 112-first group of camera indicator lights, 113-second group of camera indicator lights, 114-first signal transmission interface, 115-second signal transmission interface, 116-external device fixing interface, 117-housing fixing interface; 200-general target tracking system, 300-field of view detection range, 400-target area, 201-image processing unit, 202-deep learning tracking module, 203-coordinate transformation module, 204-real-time display module, 205-external device interface, 206-external device, 207-real-time display unit. DETAILED DESCRIPTION

[0022] The goal of radiotherapy is to accurately deliver radiation (electrons, photons, protons, heavy ions, etc.) to the irradiation target area while minimizing radiation damage to surrounding normal tissues. Accurate positioning of the irradiation target area is a key to the successful implementation of radiotherapy. For irradiation targets with relatively fixed shapes and positions, high irradiation accuracy can be achieved as long as they are well fixed and positioned before irradiation. For moving targets, their positions and shapes change over time, which brings great challenges to radiotherapy. Affected by the interaction between the target area movement and the dynamic irradiation process of radiation, the dose distribution in the target area is distorted, and the surrounding normal tissues are damaged by radiation, which seriously affects the efficacy of radiotherapy. At present, there are many motion management methods applied to actual treatment, such as: gating method, active tracking method, etc., which can greatly improve the accuracy of radiation irradiation by irradiating the target area relative to the target area during specific movement or realizing the follow-up irradiation of the target area according to the real-time motion position of the target area. However, the implementation of these motion management methods depends on the accurate detection of the real-time position of the target area to determine the current motion phase of the target area and the position of the radiation to be irradiated, which is also the key technology for accurate radiotherapy of moving targets.

[0023] Precise detection of target area motion is a key technology for achieving precise radiotherapy of moving target areas. In view of the problems of poor accuracy, low efficiency, damage, need for markers, complex operation, and high cost in existing target area motion detection technology, the present invention proposes a body surface tracking device in radiotherapy, a general target tracking system and method based on deep learning, which can directly track surface target points and internal target areas without placing tracking targets such as reflective balls on the body surface, or implanting metal markers in the body. At the same time, the trained deep learning tracking model is universal and suitable for tracking surface and internal target areas in any scenario. It can solve the defects of existing motion detection technology, realize low-cost, high-precision, efficient, intelligent, and convenient real-time detection and tracking of moving target areas, and provide guarantee for precise radiotherapy of moving target areas.

[0024] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0026] Embodiment 1: In this embodiment, a body surface tracking device 100 for radiotherapy is provided. Figure 1 As shown, the device comprises: The housing 101 is used to package the components of the device and also serves to be aesthetically pleasing and dustproof; The first imaging device 102 and the second imaging device 103 are symmetrically arranged back to back and located in the middle of the housing 101 , and a first moving component 104 and a second moving component 105 are respectively arranged at the bottom of the first imaging device 102 and the second moving component 105 .

[0027] The first moving assembly 104 and the second moving assembly 105 are respectively disposed at the bottom of the housing 101 to drive the corresponding first imaging device 102 and the second imaging device 103 to move.

[0028] The first optical reflector 106 and the second optical reflector 107 are respectively arranged at the two ends of the shell 101 through the third moving component 108 and the fourth moving component 109, and the third moving component 108 and the fourth moving component 109 are respectively arranged on the two end side walls of the shell 101, the third moving component 108 adjusts the inclination angle of the first optical reflector 106, and the fourth moving component 109 adjusts the inclination angle of the second optical reflector 107; the body surface image is reflected to the imaging device through the two optical reflectors, so there is no need to point the lens of the imaging device directly at the body surface, thereby reducing the size of the entire device and making the entire device more compact.

[0029] Directly below the first optical reflector 106 and the second optical reflector 107 , a first transmission window 110 and a second transmission window 111 are respectively provided at the bottom of both ends of the housing 101 to transmit external light signals into the housing 101 .

[0030] In the above embodiment, the first motion component 104 and the second motion component 105 are used to adjust the relative position and angle between the first imaging device 102 and the second imaging device 103, so that the two imaging devices can focus on the body surface to be collected at the same time. The third motion component 108 and the fourth motion component 109 can rotate, and the purpose of focusing on different collection surfaces is achieved by adjusting the angle between the imaging device and the optical reflector.

[0031] In the above embodiment, both the first imaging device 102 and the second imaging device 103 use visual cameras to collect body surface image information from two angles in real time, and their pixel resolution and image acquisition frequency can be selected according to actual requirements for position accuracy and time resolution.

[0032] In the above embodiment, two groups of camera indicator lights are arranged in the middle of the housing 101, wherein the first group of camera indicator lights 112 is connected to the first imaging device 102, and the second group of camera indicator lights 113 is connected to the second imaging device 103, for indicating the current working status of the camera.

[0033] In the above embodiment, a first signal transmission interface 114 and a second signal transmission interface 115 are also provided in the middle of the housing 101. The two signal transmission interfaces are used to power the visual camera on the one hand, and transmit the images collected by the visual camera in real time to the image processing device on the other hand.

[0034] An external device fixing interface 116 is provided in the middle of the housing 101 , which can connect the device of the present invention to an external device, so as to be used for motion signal collection in different scenes. Housing fixing interfaces 117 are provided at the side walls at both ends of the housing 101 .

[0035] Embodiment 2, in this embodiment, as Figure 2 As shown, a general target tracking system 200 based on deep learning in radiotherapy is provided, which is implemented based on the body surface tracking device 100 in radiotherapy in Example 1 (the device is simplified as a body surface tracking device in the following description), and is used for real-time position detection of the target area on the body surface and in the body in three-dimensional space for the moving target area in radiotherapy by establishing a general target tracking model and tracking system based on deep learning. In this embodiment, the general target tracking system 200 includes: The body surface tracking device 100 is used to collect body surface motion images in real time, and its field of view position and range are adjustable; The image processing unit 201 receives the synchronous image information collected by the two sets of visual cameras of the body surface tracking device 100 in the field of view detection range 300 in real time, and inputs it into the deep learning tracking module 202, synchronously outputs the pixel coordinates of the target area 400 on the two sets of visual camera images respectively, converts the pixel coordinates into three-dimensional space coordinates through the coordinate transformation module 203, and transmits the relevant information to the real-time display module 204 for visual display, and sends the image or coordinate information to the external device 206 through the external device interface 205; The real-time display unit 207 is connected to the image processing unit 201, and visually displays the image information, deep learning tracking area, real-time pixel coordinates and three-dimensional space coordinates, and other auxiliary information transmitted by the real-time display module 204 in the image processing unit 201.

[0036] In the above embodiment, in the deep learning tracking module 202 of the image processing unit 201, as Figure 3 As shown, the deep learning tracking process specifically includes the following steps: 101) selecting a body surface tracking model and training the body surface tracking model, and the body surface tracking model is a neural network model with a self-attention mechanism; In this embodiment, the body surface tracking model can adopt the Transformer model or other visual models, and use a large data set (not limited to body surface images) to train the model so that it has accuracy, efficiency, stability and versatility, and can track body surface movements in any scenario.

[0037] 102) A tracking area is selected on the real-time acquired image, and a trained body surface tracking model is input, and the body surface tracking model tracks the selected area in real time to output the real-time pixel coordinate position of the outer frame or outline of the tracking area.

[0038] In the above embodiment, in the image processing unit 201, pixel coordinates are converted into three-dimensional space coordinates, which specifically includes the following steps: 103) Two sets of visual cameras are used to simultaneously capture images of the coordinate calibration plate at multiple angles and positions, and the internal and external parameter matrices of each camera are obtained through a spatial camera calibration method. The parameter matrix is ​​used to convert the real-time pixel coordinates of the external frame or contour of the tracking area into three-dimensional space coordinates and output.

[0039] Embodiment 3: This embodiment provides a method for tracking a target region in vivo without labeling. The method is implemented based on the general target tracking system based on deep learning in radiotherapy in the above embodiment 2. Specifically, Figure 4 As shown, the marker-free in vivo target area tracking method in this embodiment includes the following steps: 201) using the transfer learning method to fine-tune and optimize the body surface tracking model, and its learning data uses in vivo imaging data; in this embodiment, the body surface tracking model is a trained body surface tracking model, which has the ability to accurately track the body surface target area in real time under any scenario, and the accuracy of the model in tracking the in vivo target area is further improved through fine-tuning and optimization.

[0040] The learning data uses in vivo imaging data, such as computed tomography (CT), digital X-ray imaging (DR), digitally reconstructed radiography (DRR), magnetic resonance imaging (MRI), and ultrasound imaging.

[0041] 202) amplifying the in vivo image data, and performing migration training on the surface tracking model using the amplified data to obtain a universal tracking model capable of tracking the target area on the surface and in the body; 203) selecting a tracking area on the in vivo image (DR, MRI, ultrasound image, etc.) acquired in real time by an external device as a target area to be tracked by the general tracking model, and inputting the area into the general tracking model; 204) The universal tracking model tracks the selected target area in real time, outputs the real-time pixel coordinate position of the target area, converts the pixel coordinates into three-dimensional space coordinates, and finally obtains the real-time coordinate position of the target area in three-dimensional space, and visualizes the relevant information, and sends the image or coordinate information to the external device through the external device interface; 205) The real-time display unit visually displays image information, deep learning tracking area, real-time pixel coordinates and three-dimensional space coordinates, and other auxiliary information.

[0042] In the above step 202), in order to improve the generalization ability of the model, the in vivo image data is augmented. In this embodiment, a deformation data augmentation method is used to augment the data, which specifically includes the following steps: 301) according to the deformation field between the image data of adjacent motion phases, a new deformation field between adjacent motion phases is obtained by a weighted summation method; In this embodiment, by calculating the deformation field between the motion phase and the reference phase, two continuous deformation fields are randomly selected and weighted superimposed to generate a new deformation field to simulate more detailed changes between the two phases.

[0043] 302) applying the new deformation field to the reference phase image data to generate virtual images between adjacent motion phases, and then performing rotation enhancement to make the motion scenes contained in the training data more comprehensive; In this embodiment, after the new deformation field is applied to the reference phase image data (ie, the base phase), a deformed image is obtained, and then a slight rotation enhancement is performed.

[0044] The present invention will be further described below by way of specific examples. Figure 5The figure shows a Transformer-based real-time tracking deep learning network structure for target area constructed by the present invention. The first part of the network is the feature extraction and target encoding part, which is responsible for extracting features from the input image sequence and encoding the target information into a form suitable for model processing. The pre-trained ResNet-101 is selected as the backbone network for feature extraction. The network can effectively extract visual features in the image, such as edge, texture, shape and other information. In order to retain the feature extraction capability of the pre-trained model, some network weights are frozen in this embodiment to prevent excessive updates during the training process. Specifically, the feature extraction part converts the input image sequence into a high-dimensional feature representation, providing a rich information basis for subsequent target encoding and prediction. After the extracted image features are combined with the spatial position encoding information, they are input into the Transformer encoder for feature encoding.

[0045] The second part of the network is the Transformer-based model prediction part, which aims to capture the long-distance dependencies between image sequences and learn effective image representations. The network consists of multiple encoder and decoder layers, each of which is equipped with a self-attention mechanism and a fully connected feedforward network to process and model the image sequence layer by layer. The main function of the encoder is to convert the input sequence into a set of context-related feature representations. In the decoder stage, in addition to receiving the output of the encoder, some special vectors (efg) are introduced as position information input to obtain the final output information. Using the Transformer encoder-decoder, more robust information can be extracted without the cumbersome regional recommendation network regression. In this embodiment, a Transformer structure including 6 encoder layers and 6 decoders is selected. Each encoder and decoder layer contains a multi-head self-attention mechanism with 8 attention heads. This means that each attention mechanism can process the input image features in parallel, thereby better capturing relevant information in the image sequence.

[0046] The last part of the model introduces a head module based on the Transformer model prediction network to classify the object and regress the bounding box. The head module consists of a classifier and a bounding box regressor, which are responsible for predicting the category and location of the object respectively. During the training process, the parameters of the head module are updated through back propagation and optimization algorithms to make the model's prediction results as close to the true annotation as possible.

[0047] After the model is trained, during the tracking process, the network will fuse the features from the reference frame and the current frame, as well as the spatial and positional codes from the previous frame, to further generate features with stronger discriminative capabilities, thereby achieving target positioning and regression rectangular boxes. In the usual tracking and positioning process, there may be situations where the image resolution is not high and it is difficult to distinguish the target and the interference of similar targets. The use of the Transformer network framework can solve the above problems to a certain extent. Since the visual Transformer can more explicitly model the relationship between elements in an image and encode the position, it can only output the feature information of the vector of interest during decoding, thereby suppressing the shielding of interfering targets and achieving accurate tracking of the target.

[0048] In body surface tracking training, the goal of the present invention is to train a universal body surface marker tracking model that can be used in various situations of patient body surface tracking. In addition to universality, there is no need to rely on special external markers, and it can be completed by simply marking with a marker. This feature makes the marking process simpler, the operation more convenient, and applicable to more situations. In view of the characteristics of body surface markers, this embodiment uses large-scale computer vision data sets COCO, GOT-10k, TrackingNet, LaSOT, etc. for network training, so that the network can converge quickly and have strong generalization capabilities. These computer vision data sets include a variety of real objects and scenes to provide a rich variety of training samples for the model. The diversity of these samples not only enriches the training data of the model, but also effectively improves the generalization ability and stability of the model.

[0049] In this embodiment, a body surface motion tracking device is constructed, such as Figure 1 As shown in the figure, an experimental parameter setting is: the camera resolution is 5 million pixels, the sampling rate is 20fps, the distance between the cameras is set to 50cm, and the working distance is about 1.5m. First, the camera is calibrated, using a rigid board with 5×6 black and white squares as the calibration board, where the size of each square is 25mm×25mm. The calibration board is photographed at multiple angles and positions by dual cameras at the same time. The stereo camera calibration method is used to obtain the internal and external parameter matrices of the two cameras, which is the basis for the conversion and identification of pixel coordinates to spatial coordinates.

[0050] The surface motion tracking test of the moving phantom under 20 motion states (motion cycle, amplitude and baseline change) was performed. During the test, a mark point was drawn on the body surface with a marker to identify the area to be tracked. The real-time dual-camera images were input into the deep learning model, and the model automatically tracked the target area and output the pixel coordinate values ​​on the two cameras. According to the camera calibration parameter matrix obtained above, the pixel coordinates were converted to three-dimensional space coordinates. Figures 6a to 6cThe figure shows the difference between the 3D coordinates obtained by device tracking and the reference coordinates in a moving state, where the red dotted line (tracking) represents the tracking position of the prototype combined with the model, and the black dotted line (NDI) represents the reference position. Figures 6a to 6c It can be seen that the two points and lines basically coincide, achieving accurate tracking of body surface movement. Figures 7a to 7c The following are the statistics of tracking errors in the x, y, and z directions under 20 motion states. Figure 7a As shown in Figure 1, the mean absolute deviation (MAD) of the x-direction tracking error is less than 0.15 mm, the MAD of 85% of the motion states is less than 0.1 mm, and the standard deviation (SD) of the tracking error is less than 0.1 mm. Figure 7b As shown in Figure 2, in the y direction, the MAD values ​​of the tracking errors in all motion states are less than 0.1 mm, and the SD values ​​are less than 0.1 mm; Figure 7c As shown, in the z direction, the MAD value of the tracking error of more than 90% of the motion states is less than 0.1mm, and the SD value is less than 0.1mm. In summary, the body surface motion tracking system and method based on deep learning of the present invention has achieved high tracking accuracy and stability, and can be used for target area motion tracking in any scenario.

[0051] For in vivo tracking, the present invention uses deformation method to perform data amplification on 25 in vivo image data (taking 4DCT as an example). The specific method and process are as follows: Figure 8 As shown in the figure. By calculating the deformation field between the motion phase and the reference phase, two continuous deformation fields are randomly selected and weighted superimposed to generate a new deformation field to simulate more detailed changes between the two phases. After the new deformation field is applied to the reference image, the deformed CT image is obtained, and then a slight rotation enhancement is performed. The deformed CT image is rotated between -3° and 3° at an interval of 1.5° for a total of four rotations to simulate the setting errors that may occur during treatment. The amplified data is processed by two-dimensional projection to obtain the digitally reconstructed image (DRR) and the projection contour of the target area on the DRR. By applying the 10,000 sets of DRR data sets obtained by amplification to transfer training for the trained body surface tracking model, the existing knowledge and experience are effectively utilized to improve the performance and generalization ability of the in vivo target area tracking model on new tasks, while reducing the dependence on a large amount of labeled data, providing an effective method and idea for the training and application of the in vivo target area tracking model.

[0052] Figure 9a , Figure 9b The following are two examples of in vivo target tracking. Figure 9a , Figure 9bIt can be seen that the tracked bounding box and the reference bounding box are basically overlapped, which proves that the in vivo target area tracking model of the present invention has high tracking accuracy. The tracking accuracy of 20 in vivo target areas is statistically analyzed, and the average absolute deviation and standard deviation of the absolute distance between the center of the tracked target area bounding box and the center of the reference bounding box are calculated, as shown in Figure 2. Fig.10 As shown in the figure, the average absolute deviation of the target area tracking in 20 cases was less than 1mm, the standard deviation was less than 0.7mm, and the average error was 0.9±0.5mm, which shows that the tracking accuracy is high and stable. The tracking time of the model is about 50ms / frame, which meets the current tracking rate requirements in radiotherapy and can achieve almost real-time tracking.

[0053] Through the construction of the above equipment and models and experimental tests, it is shown that the general target tracking system and method based on deep learning of the present invention can realize the accurate, efficient, non-destructive, label-free, convenient and low-cost real-time detection and tracking of the three-dimensional spatial position of target areas on the body surface and inside the body, thereby providing strong support for precise radiotherapy of moving target areas.

[0054] In addition to the tracking model based on the Transformer framework in the specific embodiment introduced here, a target positioning method based on target matching, a target positioning method based on a convolutional neural network, a target positioning method based on a regional convolutional neural network, and a target positioning method based on a U-net network can also be used to accurately output the pixel coordinates of the target area on the image.

[0055] In addition, the system and method of the present invention can not only be used for tracking surface motion and internal motion, but can also be used for surgical navigation, motion monitoring of equipment in the treatment room, and motion monitoring in other scenarios.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A general target tracking system based on deep learning in radiotherapy, characterized in that: include: The body surface tracking device in radiotherapy is used to collect body surface motion images in real time, and its field of view position and range are adjustable; The body surface tracking device in radiotherapy comprises a first imaging device (102) and a second imaging device (103), wherein both the first imaging device (102) and the second imaging device (103) use visual cameras for collecting body surface image information in real time from two angles; An image processing unit receives in real time the synchronous image information collected by the two sets of visual cameras of the body surface tracking device within the field of view detection range, and inputs it into the deep learning tracking module, synchronously outputs the pixel coordinates of the target area on the two sets of visual camera images, converts the pixel coordinates into three-dimensional space coordinates through the coordinate transformation module, and transmits the relevant information to the real-time display module for visual display, and sends the image or coordinate information to the external device through the external device interface; The real-time display unit is connected to the image processing unit and visually displays the image information, deep learning tracking area, real-time pixel coordinates and three-dimensional space coordinates transmitted by the real-time display module in the image processing unit.

2. The general target tracking system based on deep learning in radiotherapy as claimed in claim 1, characterized in that: Body surface tracking devices used in radiotherapy include: Housing (101); The first imaging device (102) and the second imaging device (103) are symmetrically arranged back to back and are located in the middle of the housing (101), and the bottoms of the first and second moving components (104 and 105) are respectively arranged; The first moving component (104) and the second moving component (105) are respectively arranged at the bottom of the housing (101) and are used to drive the corresponding first imaging device (102) and second imaging device (103) to move; The first optical reflector (106) and the second optical reflector (107) are respectively arranged at two ends of the housing (101) through a third moving component (108) and a fourth moving component (109), and the third moving component (108) and the fourth moving component (109) are respectively arranged on two end side walls of the housing (101), the third moving component (108) is used to adjust the tilt angle of the first optical reflector (106), and the fourth moving component (109) is used to adjust the tilt angle of the second optical reflector (107); Located directly below the first optical reflector (106) and the second optical reflector (107), a first transmission window (110) and a second transmission window (111) are respectively provided at the bottom of both ends of the housing (101) to transmit external light signals into the interior of the housing (101).

3. The general target tracking system based on deep learning in radiotherapy as claimed in claim 2, characterized in that: A first group of camera indicator lights (112) connected to the first imaging device (102) and a second group of camera indicator lights (113) connected to the second imaging device (103) are arranged in the middle of the housing (101); A first signal transmission interface (114) and a second signal transmission interface (115) are provided in the middle of the housing (101); an external device fixing interface (116) is also provided in the middle of the housing (101), and housing fixing interfaces (117) are respectively provided on the side walls at both ends of the housing (101).

4. The general target tracking system based on deep learning in radiotherapy according to claim 1, characterized in that: In the deep learning tracking module of the image processing unit, the deep learning tracking process includes: A body surface tracking model is selected and trained, wherein the body surface tracking model is a neural network model with a self-attention mechanism; A tracking area is selected on the real-time acquired image, and a trained body surface tracking model is input. The body surface tracking model tracks the selected area in real time to output the real-time pixel coordinate position of the outer frame or outline of the tracking area.

5. The general target tracking system based on deep learning in radiotherapy according to claim 1, characterized in that: The body surface tracking model uses the Transformer model and uses a large data set to train the model to track body surface movements in any scenario.

6. The general target tracking system based on deep learning in radiotherapy according to claim 1, characterized in that: Convert pixel coordinates to three-dimensional space coordinates, including: Two sets of visual cameras are used to simultaneously capture images of the coordinate calibration plate at multiple angles and positions. The internal and external parameter matrices of each camera are obtained through the spatial camera calibration method. The parameter matrix is ​​used to convert the real-time pixel coordinates of the external frame or outline of the tracking area into three-dimensional space coordinates and output.

7. A marker-free in vivo target tracking method, implemented based on the general target tracking system based on deep learning in radiotherapy as claimed in any one of claims 1 to 6, characterized in that: include: The body surface tracking model is fine-tuned and optimized using transfer learning methods, and the learning data is in vivo imaging data; The in vivo imaging data is amplified, and the amplified data is used to transfer the surface tracking model to obtain a universal tracking model that can realize the tracking of the target area on the surface and in the body; Selecting a tracking area on the in-vivo image acquired in real time by an external device as a target area to be tracked by the general tracking model, and inputting the target area into the general tracking model; The general tracking model tracks the selected target area in real time, outputs the real-time pixel coordinate position of the target area, converts the pixel coordinates into three-dimensional space coordinates, and finally obtains the real-time coordinate position of the target area in three-dimensional space, and visualizes the relevant information. At the same time, the image or coordinate information is sent to the external device through the external device interface.

8. The label-free in vivo target tracking method according to claim 7, characterized in that: The in vivo imaging data is augmented by using a deformation data augmentation method.

9. The label-free in vivo target tracking method according to claim 8, characterized in that: Deformation data augmentation methods include: According to the deformation field between the image data of adjacent motion phases, a new deformation field between adjacent motion phases is obtained by weighted summation method; The new deformation field is applied to the reference phase image data to generate virtual images between adjacent motion phases, and then rotation enhancement is performed.

Images