Dual-domain tracking of target structures

Abstract

The embodiments described herein provide a probability distribution for determining the matching of 3D points in a template feature map with 3D points in space. A dual-domain target structure tracking end-to-end system receives one-dimensional or two-dimensional projection data and a 3D simulated image. The end-to-end system extracts a template feature map from the simulated image using segmentation. The end-to-end system extracts (206) features from the projection data, transforms the features of the projection data into a 3D space, and sorts the 3D space to generate a 3D feature map. The end-to-end system compares the template feature map with the generated 3D feature map (210) to determine (212) the instantaneous probability distribution of the template feature map appearing in the 3D feature map.

Description

Technical Field

[0001] This application generally involves using artificial intelligence modeling to determine the location of a target structure in three-dimensional space. Background Technology

[0002] Radiation therapy (radiation-based therapy) is used to treat cancer by delivering high doses of radiation that can kill cells or shrink tumors. The target area of ​​a patient's anatomical structure intended to receive radiation (e.g., a tumor) is called the planned target volume (PTV). The goal is to deliver sufficient radiation to the PTV during radiation therapy to kill cancer cells. However, other organs or anatomical regions adjacent to or surrounding the PTV may be present in the form of a radiation beam and may receive sufficient radiation to damage or harm such organs or anatomical regions. These organs or anatomical regions are called organs of risk (OARs). Typically, physicians or radiologists identify the PTV and OAR before radiation therapy using images such as computed tomography (CT) images, cone-beam computed tomography (CBCT) images, four-dimensional CT images (e.g., CT images over time), magnetic resonance imaging (MRI) images, positron emission tomography (PET) images, ultrasound images, images obtained via some other imaging modalities, or combinations thereof. Additionally, simulated images of the patient's anatomy are obtained (using the various imaging modalities discussed herein). Physicians or radiologists may manually label the PTV and / or OAR on simulated images of the patient's anatomy to generate planning images.

[0003] In an ideal imaging system, radiation rays travel along a corresponding straight path from the radiation source through the patient to the corresponding pixel detectors of the imaging system to generate projection data. Imaging (or scanning) the patient's internal structures generates projection data to identify the PTV and / or OAR. Throughout the entire process of radiotherapy setup (e.g., during patient setup, after patient setup, during imaging of the patient, and / or while the patient is being treated with radiation), the patient's anatomy, and especially the PTV and / or OAR, can move. Different imaging modalities may require different amounts of time to complete. The patient may move during the imaging process, and especially during imaging processes that require time to complete. In the example, a CBCT scan takes longer than a CT scan. Therefore, the patient (or the PTV and / or OAR within the patient) may move during (or after) a CBCT scan and before the patient is treated with radiation. Patient movement during (or after) a CBCT scan can lead to inaccurate indications of the PTV and / or OAR within the patient.

[0004] Several conventional methods provide mechanisms for determining the location (or tracking) of individual PTVs in an image using projective scanning. However, these methods are limited by heuristically performing preprocessing steps (e.g., enhancing contrast, reducing noise) or heuristically determining the features, parameters, and / or algorithms for PTV tracking. For example, a user can perform template extraction to manually tune a region of interest (such as a PTV), thereby creating a template image of the region of interest to be used during 3D localization. Manual template extraction may include preprocessing steps such as segmentation or generating structural data (such as PTV data) from a simulated image. The preprocessed template image can be ingested by a system performing PTV tracking (e.g., determining the position of the PTV in space). However, these conventional methods are undesirable because human error and / or bias can lead to large uncertainty margins associated with PTV and / or OAR tracking. Furthermore, conventional methods for tracking systems are limited because they locate the PTV and / or OAR in two-dimensional space.

[0005] Furthermore, conventional reconstruction algorithms are limited because they often fail to accurately estimate the motion of the PTV and / or OAR when the motion is slow and the imaged area mostly has low-contrast features (as is often the case in CBCT abdominal imaging). Incorrect estimation of PTV and / or OAR motion increases the uncertainty margin associated with the PTV and / or OAR, thereby increasing the likelihood of damage to the OAR. Summary of the Invention

[0006] For the reasons described above, there is a need to create an end-to-end system for automatically and seamlessly tracking and identifying (multiple) PTVs and / or (multiple) OARs. As discussed herein, target structure tracking systems can track (multiple) PTVs and / or (multiple) OARs over time in three-dimensional space based on kV projection (or other projections) and / or based on two-dimensional projected images. This document discloses systems and methods capable of overcoming the aforementioned drawbacks, which can provide any number of additional or alternative benefits and advantages. For example, by implementing the systems and methods described herein, the system can reduce the uncertainty margin associated with manually tuned algorithms. The embodiments described herein provide the probability of determining the instantaneous three-dimensional (3D) position of target structures, such as various PTVs and / or OARs, using an end-to-end target structure tracking system.

[0007] In one aspect, the present invention provides a computer-implemented method for position prediction using an end-to-end target structure tracking system. In another aspect, the present invention provides a system.

[0008] The machine learning model used in the end-to-end system described in this paper can learn during training to extract features from projection data and select relevant features for projection based on its ability to determine the location of PTV and / or OAR structures in 3D space. The machine learning model can predict the 3D localization of the target structure based on the backprojection features of the acquired 2D projection images. The end-to-end system can also learn optimal weights for weighted logarithmic subtraction of two-energy projections during training based on image features that are well-suited for predicting the location of PTV and / or OAR structures. The end-to-end system can also learn the contrast-to-noise ratio of the structure in 3D space during training. The end-to-end system can also learn relevant features and / or segmentation information during training, which will be extracted from simulated (or planned) images based on the extent to which the end-to-end system is suitable for determining the location of PTV and / or OAR structures in 3D space. After tuning during training, the end-to-end system can be deployed in testing. During inference (sometimes called prediction or testing), an end-to-end target structure tracking system can use learned relevant features, learned algorithm weights, learned parameters, and learned functionality to generate an accurate probability distribution of the location of PTV and / or OAR structures in 3D space.

[0009] In one embodiment, a computer-implemented method for location prediction using an end-to-end target structure tracking system includes: executing a machine learning model by a computer to extract a feature set from imaging projection data associated with a target structure of a patient's anatomy; executing a reconstruction algorithm by a computer to transform the extracted feature set into a feature set in three-dimensional space; executing a recurrent neural network by a computer to obtain three-dimensional feature map data associated with the target structure, the recurrent neural network being configured to sort the imaging projection data using the feature set in three-dimensional space; extracting a template feature map from a three-dimensional simulated image by a computer, the template feature map including the target structure; comparing the template feature map with three-dimensional image data by a computer; and indicating by a computer the probability that a three-dimensional point in the template feature map matches a three-dimensional point in the three-dimensional feature map data.

[0010] In another embodiment, a system includes: a server, including a processor and a non-transient computer-readable medium containing instructions that, when executed by the processor, cause the processor to perform operations including: executing a machine learning model to extract a set of features from imaging projection data associated with a target structure of a patient's anatomy; executing a reconstruction algorithm to transform the extracted feature set into a set of features in three-dimensional space; executing a recurrent neural network to obtain three-dimensional feature map data associated with the target structure, the recurrent neural network being configured to rank the imaging projection data using the set of features in three-dimensional space; extracting a template feature map from a three-dimensional simulated image, the template feature map including the target structure; comparing the template feature map with the three-dimensional feature map data; and indicating the probability that the positions of three-dimensional points in the template feature map match those of three-dimensional points in the three-dimensional feature map data. Attached Figure Description

[0011] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and not intended to be drawn to scale. Unless indicated as background art, the drawings illustrate various aspects of the present disclosure.

[0012] Figure 1 The illustration shows the components of a target structure tracking system according to an embodiment.

[0013] Figure 2 The execution steps of a method for determining the probability of a structure's location in 3D space, according to an embodiment, are shown.

[0014] Figure 3 The illustration shows a target structure tracking system according to an embodiment for determining the probability distribution of the location of a structure in 3D space.

[0015] Figure 4 The illustration shows the use of supervised learning to train a model according to an embodiment.

[0016] Figure 5 A simplified neural network model according to an embodiment is illustrated.

[0017] Figure 6 The illustration shows a target structure tracking system according to an embodiment for determining the probability distribution of the location of a structure in 3D space.

[0018] Figure 7 The illustration shows a target structure tracking system according to an embodiment for determining the probability distribution of the location of a structure in 3D space.

[0019] Figure 8 The illustration shows a target structure tracking system according to an embodiment for determining the probability distribution of the location of a structure in 3D space.

[0020] Figure 9The illustration shows a planned image received by a target structure tracking system according to an embodiment.

[0021] Figure 10 The illustration shows a visual probability distribution of the structural locations in a 2D image of a 3D space according to an embodiment. Detailed Implementation

[0022] Referring now to the illustrative embodiments depicted in the accompanying drawings, and these illustrative embodiments will be described herein in specific language. However, it is to be understood that this is not intended to limit the scope of the claims or this disclosure. Changes and further modifications to the inventive features illustrated herein, as well as additional applications of the principles of the subject matter illustrated herein, will be considered within the scope of the subject matter disclosed herein, and will occur to those skilled in the art and those possessing this disclosure. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of this disclosure. The illustrative embodiments described in the detailed description are not intended to limit the subject matter presented.

[0023] Radiation therapy clinics can utilize software solutions to perform radiation therapy. These solutions can analyze current imaging information (e.g., real-time projection data), utilize temporal information about patient movement, and analyze radiation therapy treatment planning data, such as historical simulation (or planning) data, to predict the location of PTV and / or OAR structures throughout the radiation therapy treatment plan.

[0024] Intelligent end-to-end target structure tracking systems can employ various combinations of machine learning models (such as neural networks), filters, algorithms, and input modalities (e.g., forward projection models and backward projection models) to determine the location of the PTV and / or OAR using projection data. End-to-end target structure tracking systems can deliver results in the form of probability distributions. These probability distributions can indicate the probability that the centroid of a specific segment (e.g., an organ, tumor, or other type of anatomical landmark) is located in a particular position in 3D space.

[0025] The target structure tracking system can operate in two domains, converting time-varying two-dimensional (2D) projection data (indicated by 2D pixels associated with positions (x,y) in the projection domain) into 3D spatial data (indicated by 3D voxels associated with positions (x,y,z) in the volume domain). For example, upon receiving a kV image, the target structure tracking system can use a reconstruction algorithm (such as back projection with optional filtering) to convert 2D projection data into 3D spatial data. Similarly, upon receiving an MRI image, the target structure tracking system can convert projection data in k-space (in the frequency domain) into 3D space (volume domain). Continuous 2D projection data and 3D spatial data are two representations of the same object / structure. Unlike conventional methods, the target structure tracking system utilizes both 3D spatial data and continuous 2D projection data by extracting relevant features from both domains. The 3D spatial data is used to determine the probability distribution of the current position of various PTVs and / or OARs in 3D space given historical projection data.

[0026] Figure 1 The illustration shows components of a target structure tracking system 100 according to an embodiment. The target structure tracking system 100 may include an analysis server 110a, a system database 110b, electronic data sources 120a to 120d (collectively referred to as electronic data sources 120), end-user devices 140a to 140e (collectively referred to as end-user devices 140), and an administrator computing device 150. Figure 1 The various components depicted may belong to a radiation therapy clinic, where, in some cases, patients may receive radiation therapy via one or more radiation therapy machines (e.g., medical device 140d) located within the clinic. These components may be interconnected via network 130. Examples of network 130 may include, but are not limited to, private or public LANs, WLANs, MANs, WANs, and the Internet. Network 130 may include wired and / or wireless communications according to one or more standards and / or via one or more transmission media.

[0027] Communication via network 130 can be performed according to various communication protocols, such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and IEEE communication protocols. In one example, network 130 may include wireless communication according to the Bluetooth specification set or another standard or proprietary wireless communication protocol. In another example, network 130 may also include communication via a cellular network, including, for example, GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), and EDGE (Enhanced Global Evolution Data) networks.

[0028] The target structure tracking system 100 is not limited to the components described herein and may include additional or other components, which are not shown for the sake of brevity and will be considered to be within the scope of the embodiments described herein.

[0029] Analysis server 110a can generate and display electronic platforms configured to use various computer models, including artificial intelligence and / or machine learning models, to identify the probabilistic locations of structures (such as PTVs and / or OARs) in 3D space. More specifically, the platform can display data trajectories and / or motion patterns of one or more PTVs and / or OARs. The electronic platform may include a graphical user interface (GUI) displayed on each electronic data source 120, end-user device 140, and / or administrator computing device 150. Examples of electronic platforms generated and hosted by analysis server 110a can be web-based applications or websites configured to be displayed on various electronic devices, such as mobile devices, tablets, personal computers, etc.

[0030] In a non-limiting example, a physician operating physician device 120b can access the platform, input patient attributes or characteristics and other data, and also instruct analysis server 110a to optimize the patient's treatment plan (e.g., segmenting simulated images or performing additional preprocessing steps on simulated (or planned) images and / or projection data captured from medical device 140d). Analysis server 110a can utilize the methods and systems described herein to automatically learn relevant features of simulated or planned images and / or projection data, and optimize predictions of the centroid (or any other part) of a specific segment (or organ, tumor, or other anatomical landmark) (such as PTV and / or OAR) at a specific 3D location. Analysis server 110a can use the probability of the instantaneous 3D location of one or more target structures to calculate the trajectory, motion, and / or deformation of the structure via one or more downstream applications. Analysis server 110a can display the results on an end-user device or adjust the configuration of one of the end-user devices 140 (e.g., medical device 140d).

[0031] Analysis server 110a can host websites accessible to users (e.g., end users) operating any electronic device described herein, where content presented via various web pages can be controlled based on each specific user's role or viewing permissions. Analysis server 110a can be any computing device, including processors and non-transient machine-readable storage devices capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, etc. While target structure tracking system 100 includes a single analysis server 110a, analysis server 110a can include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0032] Analysis server 110a can execute software applications configured to display electronic platforms (e.g., hosted websites), which can generate various web pages and provide them to each electronic data source 120 and / or end-user device 140. Different users can use the website to view and / or interact with the prediction results. Different servers (such as server 120c and clinic server 140c) can also use the prediction results in downstream processing. For example, analysis server 110 can use the probability of the instantaneous 3D position of one or more target structures to track the movement of one or more structures over time. The probability distribution received by analysis server 110a (or server 120c and / or clinic server 140c) from the target structure tracking system can be applied, for example, to cardiac radiofrequency ablation, making it possible to identify patterns of beating hearts, thereby minimizing damage to healthy tissue. Additionally or alternatively, target structure tracking system 100 can track soft tissue structures (e.g., tumors or OARs) based on kV projection data.

[0033] Analysis server 110a can be configured to require user authentication based on a set of user authorization credentials (e.g., username, password, biometrics, password credentials, etc.). Analysis server 110a can access system database 110b, which is configured to store user credentials, and analysis server 110a can be configured to reference those user credentials to determine whether the set of credentials entered (purely for user authentication) matches the appropriate set of credentials that identifies and authenticates the user.

[0034] The analytics server 110a can also store data associated with each user operating one or more electronic data sources 120 and / or end-user devices 140. The analytics server 110a can use the data to weigh interactions and train various AI models accordingly. For example, the analytics server 110a can indicate that a user is a medical professional, whose input can be monitored and used to train the machine learning or other computer models described herein.

[0035] Analysis server 110a can generate and host web pages based on specific user roles within system 100. In this implementation, a user's role can be defined by data fields and input fields in user records stored in system database 110b. Analysis server 110a can authenticate users and identify user roles by executing access directory protocols (e.g., LDAP). Analysis server 110a can generate web page content tailored to the user roles defined in the user records in system database 110b.

[0036] Analysis server 110a can receive simulation (or patient setup or plan) data (e.g., historical simulation images and preprocessed segments) from a user, or retrieve such data from a data repository, analyze the data, and display the results on an electronic platform. For example, in a non-limiting example, analysis server 110a can query and retrieve simulation images from database 120d and combine the simulation images with segmented data received from a physician operating physician device 120b. Analysis server 110a can then analyze the retrieved data using various models (stored within system database 110b). Analysis server 110a can then display the results via an electronic platform on administrator computing device 150, e-physician device 120b, and / or end-user device 140.

[0037] Electronic data source 120 can represent various electronic data sources that contain, retrieve, and / or input data associated with a patient's treatment plan, including patient data and treatment data. For example, analysis server 110a can use clinic computer 120a, physician device 120b, server 120c (associated with physicians and / or clinics), and database 120d (associated with physicians and / or clinics) to retrieve / receive data associated with a patient's treatment plan.

[0038] End-user device 140 can be any computing device, including processors and non-transient machine-readable storage media capable of performing the various tasks and processes described herein. Non-limiting examples of end-user device 140 may include workstation computers, laptop computers, tablet computers, and server computers. In operation, various users can use end-user device 140 to access a GUI managed by the analysis server 110a. Specifically, end-user device 140 may include clinic computer 140a, clinic database 140b, clinic server 140c, medical devices (such as CT scanners, radiotherapy machines (e.g., linear accelerators, particle accelerators (including circular accelerators) or cobalt machines, etc. (140d)), and clinic device 140e.

[0039] Administrator computing device 150 may represent a computing device operated by a system administrator. Administrator computing device 150 may be configured to display data processing attributes generated by analytics server 110a (e.g., various analytical metrics determined during the training of one or more machine learning models and / or systems); monitor various models utilized by analytics server 110a, electronic data source 120, and / or end-user device 140; review feedback; and / or facilitate the training or retraining of neural networks maintained by analytics server 110a.

[0040] The medical device 140d may be a radiotherapy machine configured to perform radiotherapy treatment on a patient. The medical device 140d may also include an imaging device capable of emitting X-rays, enabling the medical device 140d to perform imaging using various methods to accurately image the patient's internal structures. For example, the medical device 140d may include a rotational imaging system (e.g., a static or rotational multi-view imaging system). Non-limiting examples of multi-view systems may include stereoscopic systems (e.g., two imaging systems may be arranged orthogonally).

[0041] Images of the patient's anatomy allow for the identification and tracking of the PTV and / or OAR. Imaging the patient's anatomy can include scanning the patient using medical device 140d with CT images, CBCT images, ultrasound images, MRI images, PET images, images obtained via some other imaging modality, or a combination thereof. Although digital tomography fusion (DTS) is not a direct tomography model, analysis server 110a can use DTS imaging to image the patient's anatomy and use the imaging data to track the PTV and / or OAR, because DTS uses the relative geometry between projections to calculate a relative 3D reconstruction with limited resolution in the imaging (e.g., depending on the scan arc angle).

[0042] The medical device 140d scans (or images) the patient's anatomical structures to generate projection data. The projection data can be 1D (e.g., a line detector in the medical device 140d rotating around the patient) or 2D (e.g., a panel detector in the medical device 140d rotating around the patient). Furthermore, the medical device 140d may be able to emit and / or generate signals of various intensities for imaging the patient's anatomical structures. For example, the projection data can be based on kV projection, MV projection, stereo kV / kV projection pairs (or projection sets), MV / kV projection pairs (or projection sets), dual-energy projection, etc. In some configurations (e.g., to support stereo kV / kV and MV / kV projections), the analysis server 110a can add an additional input layer to the target structure tracking system. If the projection is a dual-energy projection, the analysis server 110a can optimize the parameters in the target structure tracking system to denoise the dual-energy image and enhance target visibility.

[0043] In operation, the analysis server 110a can receive projection data (1D or 2D) from the medical device 140d. The analysis server can use a machine learning model in the target structure tracking system 100 to extract feature maps from the projection data. The analysis server 110a can convert the feature maps to different dimensions using, for example, differentiable back projection layers or any other suitable method to convert 2D data to 3D data (e.g., 3D tomographic reconstruction, iterative reconstruction algorithms, manifold learning, etc.).

[0044] Analysis server 110a can also receive 3D simulated (or planned) images, and in some configurations, can also receive segmentation information. The segmentation information can be a depiction of structures (e.g., PTVs and / or OARs) in the simulated image. Analysis server 110a can use a machine learning model in target structure tracking system 100 to generate 3D template images from the simulated images and segmentation information. In some configurations, the template image can be a human-readable image. In some configurations, the feature map can be represented by a human-readable image. In some configurations, the template image can represent relevant features used for matching and / or tracking PTVs and / or OARs. For example, the template image can be a feature map. Analysis server 110a can use various methods to generate the template image, such as the methods and systems described in U.S. Patent Publication No. 2020 / 0285915, which are incorporated herein by reference in their entirety. Analysis server 110a can compare the template images (or feature maps) to generate a probability distribution of the location of structures such as PTVs and / or OARs (indicated by the template image) in 3D space.

[0045] Analysis server 110a can communicate with medical device 140d (in real-time or near real-time), enabling the server / computer hosting medical device 140d to adjust the medical device 140d based on treatment attributes generated by analysis server 110a. For example, a radiotherapy machine can use the probabilistic location of the PTV and / or OAR structures determined by analysis server 110a to adjust the gantry, beam blocking devices (e.g., multi-leaf collimator MLC), and treatment table based on the trajectory of the PTV and / or OAR. Analysis server 110a can transmit instructions to the radiotherapy machine indicating any number or type of treatment attributes (e.g., field geometry settings) to facilitate such adjustment.

[0046] Analysis server 110a may store machine learning models (e.g., neural networks, random forests, support vector machines, or other deep learning models) trained to predict the probability that the centroid (or other part) of a specific segment (or organ, tumor, or other anatomical landmark) lies in a 3D space. Analysis server 110a may also store target structure tracking systems (e.g., chains of machine learning models and other algorithms, filters, etc.) trained to predict the probability that the centroid of a structure lies in a 3D space. The target structure tracking systems trained and stored may include systems with multi-channel inputs and multi-channel outputs. Depending on the received input (e.g., dual-energy input versus single-energy input), analysis server 110a may apply a system configured to receive multi-channel inputs or a system configured to receive single-channel inputs.

[0047] The machine learning model stored in system database 110b can correspond to a single radiation clinic or other different sets of radiation therapy machines (e.g., located in a single radiation clinic, located in a different geographical area, treating a specific type of disease (e.g., different types of cancer), and / or treating a specific gender, etc.). For example, both the machine learning model and the end-to-end target structure tracking system can be associated with identifiers indicating a radiation clinic, a set of radiation therapy machines, or a specific disease, and the machine learning model and the end-to-end target structure tracking system are configured to predict the probability of the location of a reference point of a template image in 3D space.

[0048] An operator at a radiation therapy clinic may access end-user device 140 located within the clinic or access an account associated with the clinic. The operator may provide input at the user interface that causes end-user device 140 to transmit a request to access a machine learning model (or end-to-end target structure tracking system) associated with the clinic and / or the radiation therapy machines located within the clinic. This request may include an identifier associated with the machine learning model, the clinic, and / or the set of radiation therapy machines, which analytics server 110a may use as a key in a lookup table to identify the machine learning model (or end-to-end system). Analytics server 110a may receive the request and, in some cases, identify the machine learning model via the identifier after authenticating the user. Analytics server 110a may transmit the identified machine learning model to end-user device 140 or send an alert indicating that the end-user device is authorized to access the model(s). Upon receiving or accessing the machine learning model and / or the end-to-end system, end-user device 140 may execute the systems and methods described herein to train or retrain the machine learning model to predict the probability of the location of a reference point in 3D space for a template image.

[0049] Figure 2The execution steps of a method 200 for determining the probability of a structure's location in 3D space according to an embodiment are shown. Method 200 may include steps 202 through 212. However, other embodiments may include additional or alternative steps, or one or more steps may be omitted entirely. Method 200 is described as being performed by, for example... Figure 1 The analysis server described herein is executed by a server such as the one described. However, one or more steps of method 200 can be performed by a server such as the one described above. Figure 1 The distributed computing system described herein can operate on any number of computing devices to execute. For example, one or more computing devices can execute locally. Figure 2 Some or all of the steps described in the document.

[0050] In step 202, the analysis server may execute a machine learning model to extract features from the projection data (e.g., data associated with the projection). In some configurations, the projection data may include time-series information. The analysis server may receive the projection data from an imaging system, such as a system capable of emitting appropriate radiation for CT images, CBCT images, MRI images, PET images, ultrasound images, etc. The projection data (e.g., imaging projection data) is generated by scanning (or imaging) the patient's anatomy using the imaging system.

[0051] Projection data can provide analytics servers (or users, downstream applications, or other servers) with image information about the internal structures of a patient's anatomy. Specifically, imaging of the patient's anatomy is performed around one or more target structures (such as PTV and / or OAR) (and projection data associated with the patient's internal structures) so that analytics servers (or users, downstream applications, or other servers) can evaluate one or more target structures (including size, location, trajectory, movement patterns, etc.).

[0052] The analysis server can use machine learning models to extract features from the projection data to generate feature maps. Depending on whether the detector used to detect radiation is a line detector or a panel detector, the projection data can be 1D or 2D. Projection data can include kV projection, MV projection, stereo kV / kV projection, MV / kV projection, two-energy projection, etc.

[0053] In some configurations, the analysis server can perform preprocessing on the projected data before extracting features. Example preprocessing algorithms may include defect pixel correction, dark field correction, conversion from transmission integral to attenuation integral (e.g., logarithmic normalization using the air norm), scattering correction, beam hardening correction, decimation, etc. In some configurations, the analysis server can extract features from the raw projected data. In some configurations, the analysis server can receive preprocessed projected data.

[0054] In step 204, the analysis server can execute a reconstruction algorithm to transform the extracted features into features in 3D space. As discussed herein, the projected data can be 1D or 2D. The analysis server can use any suitable 3D reconstruction method, such as the Feldkamp-Davis-Kress (FDK) algorithm, to transform 1D or 2D projected data into 3D spatial data.

[0055] In step 206, the analysis server may execute a machine learning model (such as a recurrent neural network) with memory (or internal state) to sort multiple features in 3D space based on multiple projections, thereby obtaining post-processed 3D feature map data. In some configurations, the analysis server may execute a recurrent neural network to sort multiple features in 3D space based on multiple projections, thereby obtaining 3D image data. The analysis server may convert 1D or 2D projection data into 3D space and subsequently sort the projection data to create a 3D image of the patient's anatomical structure. In some configurations, the 3D image may be a human-readable image. However, the 3D image is not limited to a human-readable image. For example, in some configurations, the 3D image may represent relevant features used for matching and / or tracking PTV and / or OAR. That is, the 3D image may be a feature map. The 3D image (and / or feature map) of the patient's anatomical structure is a simulation of the patient's anatomical structure / organ that includes the target structure in 3D space.

[0056] In step 208, the analysis server can extract a template-post-processed 3D feature map from diagnostic images, treatment simulation images, treatment plan images, or patient setup images. In some configurations, the analysis server can extract a template image from diagnostic images, treatment simulation images, treatment plan images, or patient setup images. Simulation (or patient setup) images are images associated with a patient's treatment. For example, an image used to diagnose a patient with a specific cancer can be referred to as a diagnostic image. In another example, a simulation image can be a CT image, 4D-CT image, CBCT image, MRI image, 4D MRI image, PET image, or other image simulating the patient's anatomy. In some configurations, a patient setup image can be a CT image, 4D-CT image, CBCT image, MRI image, 4D MRI image, PET image, or other image of the patient's anatomy taken during a patient setup. Therefore, extraction is not limited to diagnostic images.

[0057] Diagnostic images, treatment simulation images, treatment planning images, or patient setup images can indicate not only a specific tumor or other disease affliction, but also the area surrounding the tumor and other structures around it. An analysis server can extract a template image (or feature map) from a simulation image (or diagnostic image or patient setup image) to generate a new image (template image or feature map) of the target tumor even without much relevant structures. The template image can be a delineated (segmented or outlined) portion of the simulation image. That is, a simulation image can become a planning image (or template image / feature map) when an outline is drawn on the image (manually or automatically).

[0058] The analysis server can receive segmented template images, eliminating the need for it to extract template images or feature maps from the simulated images (or patient setup images). In some configurations, the analysis server can receive both simulated images (or patient setup images) and segmentation information. The analysis server can then use the segmentation information to extract template images (or feature maps) from the simulated images (or patient setup images).

[0059] In step 210, the analysis server may compare the template-post-processed 3D feature map with the post-processed 3D feature map. In some configurations, the analysis server may compare the template image with 3D image data (e.g., the 3D image data determined in step 206). In some configurations, the analysis server may use any suitable means for template matching. The analysis server may apply a correlation filter to determine the correlation between the template image and the 3D image data. The analysis server may perform feature-based comparisons of the template image and the 3D image (e.g., using a neural network). The analysis server may convolve the template image and the 3D image data to evaluate the similarity between the template image and the image. In operation, the analysis server may compare one or more points in the template image with one or more points in the 3D image to determine whether a point in the 3D image matches a reference point in the template image. The analysis server may determine whether (and where) the template image is represented in the 3D image or the probability that the template image (or features) is located in the 3D image.

[0060] In step 212, the analysis server can generate a probability distribution indicating the probability that a reference point in the template-post-processed 3D feature map matches a point in the post-processed 3D feature map. In some configurations, the analysis server can compare the centroids (or portions) of structures in the post-processed 3D feature map (or template image) that match points in the post-processed 3D feature map (or 3D image). Based on the comparison in step 210, the probability map can be transmitted by the analysis server to a downstream application (e.g., a subsequent algorithm) that performs analysis indicating peaks (e.g., peaks of sidelobes) indicating a match to the target structure. The analysis server can also transmit the uniqueness or confidence of such a match to the downstream application.

[0061] Figure 3 The illustration shows a target structure tracking system 300 according to an embodiment for determining the probability of the instantaneous 3D position of one or more target structures in 3D space. The target structure tracking system 300 includes a feature extractor model 304, a 3D reconstruction model 306, a machine learning model 308 with memory, a template extractor model 314, and a matching 310. The matching 310 may be a matching protocol, a matching operation, a comparator performing the matching, or a matching method and system described in U.S. Patent No. 8,396,248 (entirely incorporated herein by reference). The target structure tracking system 300 may optionally include a 2D forward projection model 318 and a segmented model 320. The inputs to the target structure tracking system 300 include a simulated image 312 (or a patient setup image) and projection data 302. The output 316 (probability distribution) is the output of the target structure tracking system 300.

[0062] The analysis server can analyze data from imaging devices (e.g., Figure 1 The medical device 140d receives 1D or 2D projection data 302. For example, the medical device (e.g., imaging equipment, radiotherapy machine) may rotate around the patient, project radiation, and detect the projected radiation to capture projection data. Projection data 302 may include kV projection, MV projection, stereo kV / kV projection, MV / kV projection, dual-energy projection, or other projection pairs (or projection sets), etc. In some configurations, the target structure tracking system 300 is capable of receiving multi-channel input and is trained to receive multi-channel input. For example, projection data 302 may be projections of the same energy from various angles. For example, the medical device may rotate around the patient, image the patient, and capture projection data 302 from various angles. Additionally or alternatively, projection data 302 may be projections from a single angle at various energies. For example, the medical device may not rotate, thus providing static imaging of the patient.

[0063] The analysis server can employ multiple machine learning models (such as neural networks, random forests, support vector machines, etc.) as feature extractor models 304 to extract feature maps from the projected data 302. In some configurations, the output of the feature extractor model 304 can be multiple feature maps jointly encoding relevant feature information. The feature information may include radiation intensity data, attenuation data, and removed data.

[0064] The analysis server can process the projection data 302 based on projection characteristics (e.g., projection intensity, projection angle, and / or the number of projections). For example, the analysis server can use a feature extractor model 304 to extract feature maps, and then use a 3D reconstruction model 306 to reconstruct the projection space into a volumetric space based on the characteristics of the projection data 302. Additionally or alternatively, the analysis server can use the 3D reconstruction model 306 to reconstruct the projection space into a volumetric space, and then use the feature extractor model 304 to extract features from the volumetric space based on the characteristics of the projection data 302. The analysis server can also apply filters to reduce noise and / or process the projection space and / or 3D space to improve the quality of the projection and / or 3D space.

[0065] Using feature extractor model 304 as part of the target structure tracking system 300 to extract feature maps has advantages over target-optimized feature extractor module 304, which tracks structures based on projection data 302. For example, features can be extracted by optimizing the weighted logarithmic subtraction of the two-energy projection based on generating image features well-suited for template image tracking. That is, feature extractor model 304 will learn how to select relevant features for tracking target structures (e.g., PTV and / or OAR).

[0066] The feature extractor model 304 can be a convolutional neural network that extracts 2D feature data and generates a 2D feature map from the 2D projected data 302. The analytics server can generate the feature map by applying a kernel to the projected data 302. For example, the kernel can slide across the input of the projected data 302, and the element-wise dot product of the kernel and the array can generate the feature map. The dimensions of the kernel and the feature map are based on the dimensions of the projected data 302. The kernel is designed to detect the presence of certain features, and the detected features are arranged in the feature map.

[0067] The analysis server can convert the feature map generated from the feature extractor model 304 from a 2D feature map into a 3D feature map, for example, in a 3D reconstruction model 306. In the example, the analysis server can employ a differentiable back projection model. A differentiable back projection model is an improvement on a filtered single projection using a reconstruction algorithm such as the FDK algorithm. That is, the analysis server can perform back projection, the FDK algorithm, etc., on a neural network level, iteratively execute the reconstruction algorithm, and use a loss function to consider the parameters learned from the projection domain (e.g., the 2D domain) and associated with the reconstruction volume domain (e.g., the 3D domain). The loss function can consider the peak-to-sidelobe ratio of the output probability distribution. The differentiable back projection model allows the analysis server to jointly optimize the corrections in the volume domain and the projection domain, rather than iteratively computing a single solution in the volume domain using the reconstruction algorithm. Therefore, the differentiable back projection model applied in the 3D reconstruction model 306 has a forward path (for 3D projection) and a backward path (to force optimization of the reconstruction algorithm over time).

[0068] The analysis server can process the resulting 3D feature maps in a machine learning model 308 with memory. Neural networks using memory may rely on hidden states to process data continuously. Neural networks using memory (such as Long Short-Term Memory (LSTM) networks or other recurrent networks) can receive hidden states as input from a database and / or memory. The analysis server can store the hidden states in the recurrent network in the form of one or a series of 3D spaces. The recurrent network continuously processes the projected data, using the current hidden state of the volume data in the LSTM (…). To predict the future hidden state of volumetric data ( In some configurations, the hidden state can be initialized using patient-set CBCT, short kV projection sequences, or synthetic projection data from previous images (e.g., simulated CT or 4D CT images).

[0069] The analysis server can use 3D feature maps to generate a 3D image for performing template matching (e.g., matching 310). The analysis server can use the hidden states as input to an LSTM, which sorts the 3D feature maps and hidden states to generate a 3D image.

[0070] In some configurations, the analysis server may employ a 2D forward projection model 318. The 2D forward projection model 318 may receive 3D feature map data from the 3D reconstruction model 306 and generate 2D projection data. The forward projection layer (or module) is used. The analysis server can perform forward projection modeling on the neural network. The forward projection model 318 can be a differentiable forward projection layer, allowing the analysis server to optimize corrections in the volume domain and projection domain during training. The data is fed into the feature extractor model 304 to emphasize relevant features of the projection. That is, the feature extractor model 304 benefits from knowledge about the 3D image domain due to relevant feature insights from the 2D projection data. The reliance on the hidden states and memory of the machine learning model 308 with memory is reduced due to the insights gained using the 2D deconstruction model 318. The analysis server learns from the data in both the 2D projection space and the 3D space. Therefore, the transformation between the 2D projection space and the 3D space is performed in a substantially lossless manner, reducing the likelihood of losing relevant features in both the 2D projection space and the 3D space.

[0071] The analysis server can extract a template image (or feature map) from the simulated image 312 (or patient setup image) using the template extractor model 314. The analysis server can extract the template image from the simulated image 312 based on a segmented structure. In some configurations, the analysis server can receive the simulated image 312 with segmented information. For example, the analysis server can receive the simulated image 312 with indicators depicting the template image on it. For example, the simulated image 312 could be of the pleural cavity and could identify tumors on the pleural cavity during preprocessing steps. The analysis server can use the segmented simulated image (e.g., a planning image) to extract the template image (or feature map).

[0072] In some configurations, the analysis server can receive both a first input (simulated image 312) and a second input (segmentation information). The user can derive the segmentation information from the radiotherapy treatment plan information associated with the simulated image 312 (and ultimately with the patient to be treated). That is, instead of segmenting the simulated image 312 according to various preprocessing steps, the analysis server applies the segmentation information to the simulated image 312 to segment it.

[0073] In some configurations, the analysis server applies a segmentation model 320 to segment the simulated image 312. The analysis server segments the simulated image 312 based on a trained segmentation model 320, which generates contours on the simulated image 312. The analysis server can learn to consider surrounding information within the simulated image 312, rather than simply cropping the image. For example, the analysis server can consider the context of a structure. If the simulated image 312 is a tumor in the pleural cavity, the analysis server can learn the features of the pleural cavity and the tumor's position relative to it. The analysis server can use reference points for the segmentation (e.g., points associated with the target structure) and learn the regions surrounding the reference points.

[0074] In some configurations, the analysis server can train a segmentation model 320 to perform multi-structure segmentation (e.g., segmentation of multiple structures in a simulated image 312). The analysis server can segment single-energy images (such as single-energy CT) and / or dual-energy images (such as dual-energy CT).

[0075] The analysis server can use template extractor model 314 to extract a template image (or feature map) based on segmentation information (automatically determined by the analysis server in segmentation model 320, or received by the analysis server) and simulated image 312. The template image can be a representation of a structure (e.g., PTV and / or OAR) used by the analysis server to match the transformed projection data 302. Template extractor model 314 can be a machine learning model (such as a neural network, deep neural network, a series of neural networks, etc.).

[0076] In some configurations, the template extractor model 314 can receive CT / CCBT images with time-series information. The template extractor model 314 can detect structural deformation, allowing the analysis server to learn how to account for deformation. In some configurations, the analysis server can employ deformation modeling to determine the deformation field associated with a target structure (or multiple target structures). Therefore, given the deformation of a structure over time, the analysis server can output a probability distribution 316 indicating the probability of a point in the template image (or feature map) at a specific location. For example, the analysis server can indicate the probability of a 3D point in the template feature map matching the location of a 3D point in the 3D feature map based on the deformation field.

[0077] The analysis server can use matching 310 to apply the 3D localization of the template generated by the template extractor model 314 to the output of the machine learning model 308 with memory. The analysis server can use matching 310 to compare the template image (or feature map) with the 3D image (or feature map). In some configurations, the analysis server can compare multiple feature maps with the 3D feature map, such that the output 316 is a multi-channel probability distribution. The analysis server can perform matching 310 using any suitable template matching method. In some configurations, matching 310 can be a 3D correlation filter. When the analysis server receives dimensionality-reduced data (e.g., preprocessed segmentation information in a simulated image), the analysis server can apply the 3D correlation filter. The correlation filter can return a correlation response indicating whether the 3D image data (from the output of the machine learning model 308 with memory) is related to the template image (or feature map) from the template extractor model 314.

[0078] In some configurations, matching 310 can be a convolution operation (or convolutional neural network) in the spatial domain. For example, a convolution operation involves convolving each point of the template image (or feature map) with each point in the 3D image (or feature map). Additionally or alternatively, the convolution operation can be performed using pointwise multiplication in the Fourier domain (e.g., using Fast Fourier Transform (FFT) and Inverse FFT (iFFT)). The analysis server can employ convolution when it receives and analyzes the complete volumetric simulation image 312. The output of the convolution operation can indicate the location of the best match between the template image and the 3D image data from the template extractor model 314.

[0079] Output 316 can be the probability of the instantaneous 3D localization of the template image in 3D space. Output 316 measures the probability that a reference point in the template image will be located at a specific 3D location in 3D space (e.g., the centroid of a specific segment / organ / tumor or any kind of anatomical landmark) (e.g., in 3D image data). Output 316 can be a probability distribution of the possible localization of the template image in 3D space. Accurate estimation of the template image's location within the patient's body improves the accuracy of other downstream algorithms and / or applications. The analysis server can deliver output 316 to downstream algorithms and / or applications, enabling the analysis server (or other servers) to determine the 3D trajectory of the structure.

[0080] The analysis server can also generate multiple probability distributions of the positions of multiple template images in 3D locations. That is, a target structure tracking system 300 can be used to track multiple template images (or feature maps). For example, the analysis server can generate a multi-channel probability distribution indicating the probability that a 3D point in a template feature map matches the position of a 3D point in a 3D image (or other feature map data), and the probability that a 3D point on a second template feature map matches the position of a 3D point in a 3D image (or other feature map data). Additionally or alternatively, the analysis server can use the multi-channel output to generate a single probability distribution of the positions of multiple template images in 3D locations. Therefore, the analysis server can simultaneously track multiple structures in 3D space using multiple channels of the target structure tracking system 300. For example, the analysis server can track OARs and PTVs.

[0081] The analysis server can also learn motion covariance between various structures. For example, as discussed in this paper, the location of the thoracic cavity and the location of a tumor within it can be determined. Due to the covariance between the tumor and the thoracic cavity, the analysis server can use the motion of the thoracic cavity when determining the motion of the tumor. In another example, the analysis server can learn motion covariance between other structures of the body, such as the duodenum and pancreas. Additionally or alternatively, the analysis server can learn offsets between various structures.

[0082] Alternatively or additionally, the target structure tracking system 300 can output a confidence value associated with the probability distribution. For example, the analysis server can evaluate the peak-to-sidelobe ratio when determining the confidence of the probability distribution. For example, the confidence value can be based on the peak and sidelobe associated with the probability distribution. Alternatively or additionally, the confidence associated with the probability distribution can be based on the structure location prediction accuracy during training.

[0083] Additionally or alternatively, the target structure tracking system 300 can output one or more classifications. For example, the analysis server can classify whether a point in the space is likely to be occupied by a target structure (e.g., a PTV and / or an OAR) based on the probability that an evaluation template image (or feature map) matches a specific location in the 3D space. The analysis server can compare the probability at a specific location with a threshold. If the probability at a specific location meets the threshold, the analysis server can classify that location as a location occupying a target structure in the 3D space.

[0084] Alternatively or additionally, the target structure tracking system 300 can output a 3D coordinate prediction of the structure's location. Alternatively or additionally, the target structure tracking system 300 can output a deformation field. That is, the target structure tracking system 300 can predict the deformation of a template image (or feature). Alternatively or additionally, the target structure tracking system 300 can output a probability distribution of the location of reference points in a template image given a structural deformation.

[0085] The analysis server can use deformation information (fields) from the target structure tracking system 300 to improve the accuracy of predicting the location of the PTV and / or OAR. For example, the analysis server can use one or more vector fields to represent deformation in 3D space. Additionally or alternatively, the analysis server can use a combination of parametric models and / or affine deformations to represent deformation in 3D space. The analysis server can use hidden states in a machine learning model 308 with memory to capture deformation information (e.g., anatomical changes in the structure).

[0086] Reference Figure 6 The analysis server can be configured with a target structure tracking system 600, enabling the feature extractor model (e.g., Figure 3 304 in the original text refers to a neural network 604 (such as a convolutional neural network). The analysis server can configure the target structure tracking system 600 to enable 3D reconstruction of the model (e.g., Figure 3 306 in the model is a differentiable back projection model 606. The analysis server can be configured with a target structure tracking system 600, enabling machine learning models with memory (e.g., Figure 3308 in the diagram is an LSTM network 608. The analysis server can also receive segments 630 (e.g., segment information from radiotherapy treatment planning information associated with the simulation image 612) and the simulation image 612. In some configurations, the simulation image 612 may be a patient setup image or a diagnostic image. The analysis server may be configured with a target structure tracking system 600 such that a template extractor model (e.g., Figure 3 314 in the template extractor is template extractor 614. The analysis server can use template extractor 614 to crop the simulated image 612 according to segmentation 630. The analysis server can configure the target structure tracking system 600 to enable matching (e.g., Figure 3 310 in the text is the correlation filter 610.

[0087] Reference Figure 7 The analysis server can be configured with a target structure tracking system 700, enabling the feature extractor model (e.g., Figure 3 The 304 in the text refers to a neural network 704 (such as a convolutional neural network). The analysis server can configure the target structure tracking system 700 to enable 3D reconstruction of the model (e.g., Figure 3 306 in the model is a differentiable back projection model 706. The analysis server can be configured with a target structure tracking system 700, enabling machine learning models with memory (e.g., Figure 3 308 in the diagram is an LSTM network 708. The analysis server can also receive segments 730 (e.g., segment information from radiotherapy treatment planning information associated with the simulated image 712) and the simulated image 712. In some configurations, the simulated image 712 may be a patient setup image or a diagnostic image. The analysis server can be configured with a target structure tracking system 700, such that a template extractor model (e.g., Figure 3 314 in the diagram is neural network 714. The analysis server can use neural network 714 to extract relevant features from simulated (or planned) images 712 and segments 730 for tracking PTV and / or OAR. The analysis server can configure the target structure tracking system 700 to enable matching (e.g., Figure 3 310 in the text is the convolution operation 710 (or convolutional neural network).

[0088] Reference Figure 8 The analysis server can be configured with a target structure tracking system 800, enabling the feature extractor model (e.g., Figure 3 The 304 in the text refers to a neural network 804 (such as a convolutional neural network). The analysis server can utilize 2D forward projection models (e.g., Figure 3The target structure tracking system 800 can be configured using a forward projection model (318) to convert 3D feature information from a differentiable back projection model 806 into 2D projection data. A neural network 804 can receive historical 2D projection data from the forward projection model 818 to extract relevant features. An analysis server can configure the target structure tracking system 800 to enable the 3D reconstruction model (e.g., ...) to... Figure 3 306 in the model is a differentiable back projection model 806. The analysis server can be configured with a target structure tracking system 800, enabling machine learning models with memory (e.g., Figure 3 308 in the diagram is an LSTM network 808. The analysis server can also receive segments 830 (e.g., segment information from radiotherapy treatment planning information associated with the simulated image 812) and the simulated image 812. In some configurations, the simulated image 812 may be a patient setup image or a diagnostic image. The analysis server can be configured with a target structure tracking system 800 such that a template extractor model (e.g., Figure 3 314 in the diagram refers to neural network 814. The analysis server can use neural network 814 to extract relevant features for tracking PTV and / or OAR. The analysis server can configure the target structure tracking system 800 to enable matching (e.g., Figure 3 310 in the text is the convolution operation 810 (or convolutional neural network).

[0089] Figure 9 The illustration shows a non-limiting example 900 of a planning image 901 received by a target structure tracking system. The planning image 901 is considered a planning image because it includes the outline (or segments) 902 around the PTV. In some configurations, the output of the target structure tracking system (e.g., a probability distribution) is transmitted to subsequent algorithms. In some configurations, the probability distribution can be displayed to the user (either as a 2D image of a 3D distribution or as a 3D distribution).

[0090] Figure 10 A non-limiting visual example 1000 is illustrated, showing the probability distribution of structural locations in a 2D image within a 3D space according to an embodiment. In this non-limiting example 1000, the target structure tracking system uses the methods described herein to determine segmented target structures (e.g., Figure 9 The probability of the target structure (902) appearing at its location in the patient's anatomical structure in 3D space. The visualized probability distribution 1001 uses color, visual patterns (e.g., cross-shading patterns), or other identifiers to indicate the probability of the target structure being located in 3D space. As shown, the target structure tracking system has determined that the target structure is likely located at location 1002.

[0091] Figure 4The illustration depicts training a machine learning model 400 using supervised learning according to an embodiment. Supervised learning is a method for training a machine learning model given input-output pairs. Input-output pairs are inputs with associated labeled outputs (e.g., expected outputs).

[0092] The analytics server can train a machine learning model 400 using supervised learning, which uses training inputs 480 (e.g., training projection data, feature maps associated with the training projection data, training simulated images, training segments, or training 3D spatial data), predicted outputs 482 (e.g., computed relevant projection features, predicted template images, predicted 3D images and / or predicted projection data, or the probability distribution of the template image in 3D space, predicted contour data, and / or predicted plan images), and expected outputs 486 (e.g., labels associated with relevant projection features, labels associated with template images, labels associated with 3D spatial data, actual 3D images and / or actual projection data, or the location of structures in 3D space, actual contour data, and / or actual plan images). The analytics server can receive input-output pairs from any data repository. The machine learning model 481 can be trained based on general data and / or granular data (e.g., patient-specific data), allowing the model to be trained specifically for a particular patient.

[0093] The analysis server can feed training inputs 480, such as training simulated images, training segments, and training projection data, into the target structure tracking system. In some configurations, only training simulated images and training projection data are used as training inputs 480. That is, the analysis server can train a segmented model (e.g., ...) in the target structure tracking system. Figure 3 (320 in the text). The analysis server can train the target structure tracking system based on the structure to be tracked in the 3D image (e.g., one structure to multiple structures) and / or the input type (e.g., multi-channel input to single-channel input based on stereo projection data).

[0094] An end-to-end target structure tracking system can be represented as a machine learning model 481. By applying the current state of the target structure tracking system to training data (e.g., simulated (or planned) training segments and training projection data), the end-to-end target structure tracking system can use training input 480 to generate a probability distribution (e.g., predicted output 482) representing the template image in the 3D space of an analysis server tracking the template structure in 3D space. The analysis server can use comparator 485 to compare the location of the structure in 3D space (e.g., expected output 486) with the predicted output 482 generated by the current state of the system to determine the amount of error or difference.

[0095] Training an end-to-end target structure tracking system may include pre-trained neural networks or other machine learning models to achieve convergent training of the end-to-end target structure tracking system. For example, an analytics server may train or pre-train each machine learning model in the end-to-end target structure tracking system before it is trained and / or used. For instance, the analytics server may feed training inputs such as training projection data 480 to models such as feature extractors (e.g., Figure 3 The feature extractor module 304 is used in the neural network. In some configurations, the analytics server can feed historical data into the neural network. The neural network can be represented by a machine learning model 481. The analytics server can use the neural network to compute relevant projective features (e.g., predicted output 482) by applying the current state of the neural network to the training projection. The analytics server can use a comparator 485 to compare the labels associated with the relevant feature projection (e.g., expected output 486, such as edge enhancement of a specific region on the projection) with the predicted output 482 computed by the current state of the neural network to determine the amount of error or difference.

[0096] The analysis server can also feed training inputs such as training simulation images and training segments into a template extractor model (e.g., Figure 3 The template extractor model 314 is used in a neural network. The neural network can be represented by a machine learning model 481. The analysis server can use the neural network to compute a predicted template image (e.g., predicted output 482) by applying the current state of the neural network to a training simulated image and training segments. The analysis server can use a comparator 485 to compare the label associated with the template image (e.g., expected output 486) with the predicted output 482 computed by the current state of the neural network to determine the amount of error or difference.

[0097] The analytics server can also feed training inputs, such as feature maps associated with the training projection data, into models such as 3D reconstruction models (e.g., Figure 3 The 3D reconstruction model 306 is used in a neural network. The neural network can be represented by a machine learning model 481. The analysis server can use the neural network to generate 3D spatial data (e.g., predicted output 482) by applying the current state of the neural network to a training projection. The analysis server can use a comparator 485 to compare the labels associated with the 3D spatial data (e.g., expected output 486) with the predicted output 482 computed by the current state of the neural network to determine the amount of error or difference.

[0098] The analysis server can also feed training inputs (such as training 3D spatial data) into neural networks (such as recurrent neural networks, LSTMs, etc.). The neural network can be a machine learning model with memory (e.g., Figure 3 The neural network can be represented by a machine learning model 481. The analysis server can use the neural network to sort 3D image data. The analysis server applies the current state of the neural network to training 3D spatial data and hidden states (e.g., historical 3D spatial data and / or historical 3D volumetric data) to predict 3D images (e.g., predicted output 482). The analysis server can use a comparator 485 to compare the actual 3D image (e.g., expected output 486) with the predicted output 482 computed by the current state of the neural network to determine the amount of error or difference.

[0099] The analytics server can also feed training inputs such as training 3D spatial data into a model such as a forward projection model (e.g., Figure 3 The neural network (such as the forward projection module 318) can be represented by a machine learning model 481. The analysis server can use the neural network to predict projection data (e.g., predict output 482, which can be 1D or 2D) by applying the current state of the neural network to 3D spatial data. The analysis server can use a comparator 485 to compare the actual projected image (e.g., the expected output 486, which can be 1D or 2D) with the predicted projection data generated by the current state of the neural network to determine the amount of error or difference.

[0100] The analysis server can also feed training inputs such as training simulation images into a segmented model (e.g., Figure 3 The segmentation module 320 in the neural network is used. The neural network can be represented by a machine learning model 481. The analysis server can use the neural network to predict contour data (e.g., predict output 482) by applying the current state of the neural network to a training simulated image. The analysis server can use a comparator 485 to compare the actual contour data (e.g., the expected output 486) with the predicted contour data generated by the current state of the neural network to determine the amount of error or difference.

[0101] Additionally or alternatively, the analysis server may use a neural network to predict a planned image by applying the current state of the neural network to a training simulated image (e.g., predicting output 482). The analysis server may use a comparator 485 to compare the actual planned data (e.g., expected output 486) with the predicted planned image generated by the current state of the neural network to determine the amount of error or difference.

[0102] During training, the error calculated by the analysis server using comparator 485 (represented by error signal 483) can be used to adjust the weights in the machine learning model 481 so that the machine learning model 481 changes (or learns) over time.

[0103] The analysis server can use, for example, a backpropagation algorithm to train an end-to-end target structure tracking system (and / or each machine learning model in the machine learning models within the end-to-end target structure tracking system). The backpropagation method operates by propagating an error signal 483. The error signal 483 can be computed in each iteration, batch, and / or epoch and propagated through all algorithm weights in the machine learning model 481, such that the algorithm weights are adapted based on the amount of error. The error is minimized using a loss function. Non-limiting examples of loss functions may include a function considering the peak-to-sidelobe ratio, a squared error function, and / or a cross-entropy error function.

[0104] An example function considering the peak-to-sidelobe ratio of a probability distribution (e.g., the probability that a reference point in a template image matches each point in 3D space) can be defined as:

[0105]

[0106] In the above equation, Y ( x () is a voxel x In reference position x The probability at 0, a It is a weighted constant that balances the peak localization error relative to the reference localization (e.g., the reference localization of the template image), and S ( Y () is the side lobe ratio. S ( Y ) can be defined as:

[0107]

[0108] Zero-centered normalized cross-correlation function Y ( x The expected value and standard deviation of ) are calculated outside the peak exclusion zone, and it is defined as:

[0109]

[0110] In the above equation, ρ TT It is the autocorrelation of the template image, and t It is the threshold of the sidelobe region.

[0111] The output of an end-to-end target structure tracking system is a probability distribution with peak probabilities at the highest points detected in the template image in 3D space. That is, the more pronounced the peak, the more accurate the tracking prediction. The loss function used to optimize the end-to-end target structure tracking system enhances the peak probability at a location in 3D space by penalizing sidelobes exceeding a threshold.

[0112] The analysis server can use one or more loss functions during the training of each neural network in an end-to-end target structure tracking system and / or a neural network. The loss function for each neural network in the end-to-end target structure tracking system can be the same or different. Introducing loss at intermediate levels (e.g., at each neural network in the neural network) can beneficially smooth the feature space (e.g., feature maps) ingested by each neural network in the neural network.

[0113] The analysis server tunes the weighting coefficients of the end-to-end target structure tracking system (or each machine learning model within the machine learning models of the end-to-end target structure tracking system) to reduce the amount of error, thereby minimizing the difference between the predicted output 482 and the expected output 486 (or otherwise converging). The analysis server can continue to feed the training input 480 into the machine learning model 481 until the error determined at the comparator 485 is within a certain threshold (or a threshold number of batches, rounds, or iterations has been reached). The analysis server can then store the trained model in a database.

[0114] The analysis server can use training input 480 received from one or more databases to train a machine learning model 481. For example, the analysis server can receive training projections and training simulation images (or patient setup images or planning images) and labels associated with the training projections and training simulation images (or patient setup images or planning images).

[0115] The labels can be the location of the 3D PTV (or OAR) within the patient's anatomy, the orientation of the 3D PTV (or OAR) within the patient's anatomy, or the segmentation or isolation of the target structure (e.g., PTV or OAR) within the patient's anatomy. Training projections and training simulation images (or patient setup images and / or planning images) can be historical projection data and historical simulation data, respectively, and the labels associated with the training projections and training simulation images (patient setup images or planning images) may have been previously determined by one or more users. The location of the structure in 3D space can be the historical location of the structure in 3D space. For each training projection, the corresponding labels can correspond to a sequence of 3D localization and / or orientation of the structure (e.g., PTV and / or OAR).

[0116] Alternatively or additionally, the training images may be historical diagnostic images (simulated images, planning images, and / or patient setup images), and the labels associated with the template images (e.g., segments) may have been previously determined by one or more users. Alternatively or additionally, the training projection data may be historical projection data, and the labels associated with the 3D spatial data may have been previously determined by one or more reconstruction algorithms. The training 3D spatial data may be historical 3D spatial data, and the labels associated with the actual 3D images may be historical 3D images of the patient.

[0117] Once trained and validated, the analytics server can perform template image tracking on unknown data (e.g., unlabeled data) during the inference phase using the target structure tracking system. The analytics server can store weights of the target structure tracking system tuned during training to minimize the loss defined by the loss function, allowing the tuned weights to be used during the inference phase. In the example, during the inference phase, the trained target structure tracking system can operate on simulated images and projected data to automatically generate a probability distribution of the position of the template image (derived from the simulated image) relative to the 3D image (generated from the projected data).

[0118] In addition to training the target structure tracking system (or machine learning model) discussed above, the analytics server can further train the target structure tracking system (or machine learning model) using user interactions. When a user performs activities on an electronic platform, the analytics server can track and record details of the user's activities. For example, when a prediction result is displayed on a user's electronic device, the analytics server can monitor the user's electronic device to identify whether the user has interacted with the prediction result by editing, deleting, accepting, or revising it. The analytics server can also timestamp each interaction, allowing it to record the frequency of modifications and the duration of revisions / corrections.

[0119] Analytics servers can utilize application programming interfaces (APIs) to monitor user activity. They can use executable files to monitor users' electronic devices. Analytics servers can also monitor the electronic platforms displayed on electronic devices via browser extensions running on those devices. Analytics servers can monitor multiple electronic devices and various applications running on them. They can communicate with various electronic devices and monitor communication between these devices and various servers running applications on them.

[0120] Using the systems and methods described herein, analysis servers can have formalized approaches to generate, optimize, and / or evaluate the probability distribution of the location of template images in 3D space within a single automated framework, based on various variables, parameters, and settings dependent on the patient and / or the patient's treatment. The systems and methods described herein enable servers or processors associated with a clinic (e.g., located within a clinic) to determine the location of (multiple) PTVs and / or (multiple) OARs within the patient's body, instead of relying on the subjective skills and understanding of technicians or physicians.

[0121] Figure 5 A simplified neural network model 500 according to an embodiment is illustrated. The neural network model 500 may include a stack of different layers (vertically oriented) that transform a variable number of inputs 502 taken in by input layer 191 into an output 506 at output layer 508. The neural network model 500 may include multiple hidden layers 510 between input layer 504 and output layer 508. Each hidden layer has a corresponding number of nodes (512, 514, and 516). In the neural network model 500, a first hidden layer 510-1 has node 512, and a second hidden layer 510-2 has node 514. Nodes 512 and 514 perform specific computations and are interconnected to nodes in adjacent layers (e.g., node 516 in output layer 508). Each node (512, 514, and 516) sums the values ​​from its neighboring nodes and applies an activation function, thereby allowing the neural network model 500 to detect nonlinear patterns in the inputs 502. Each node (512, 514, and 516) is interconnected by weights 520-1, 520-2, 520-3, 520-4, 520-5, and 520-6 (collectively referred to as weights 520). Weights 520 are tuned during training to adjust the strength of the nodes. Adjusting the node strength enhances the neural network's ability to predict accurate outputs 506. When the analytics server trains the target structure tracking system in an end-to-end manner, the analytics server transfers the system from the input layer of the target structure tracking system (… Figure 3 The projected data 302 (input) and the simulated image 312 (input) are trained to the output layer of the system. Figure 3 Output 316 in the middle.

[0122] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure or the claims.

[0123] Implementations in computer software can be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. Code segments or machine-executable instructions can represent procedures, functions, subroutines, programs, routines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments can be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0124] The actual software code or dedicated control hardware used to implement these systems and methods does not limit the claimed features or this disclosure. Therefore, the operation and behavior of the systems and methods are described without reference to specific software code, and it is to be understood that the software and control hardware can be designed to implement the systems and methods based on the description herein.

[0125] When implemented in software, functionality can be stored as one or more instructions or code on a non-transient computer-readable or processor-readable storage medium. The steps of the methods or algorithms disclosed herein can be implemented in a processor-executable software module that may reside on a computer-readable or processor-readable storage medium. Non-transient computer-readable or processor-readable media include computer storage media and tangible storage media that facilitate the transfer of computer programs from one place to another. Non-transient processor-readable storage media can be any available medium accessible to a computer. By way of example, and not limitation, such non-transient processor-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other tangible storage medium that can be used to store desired program code in the form of instructions or data structures and is accessible to a computer or processor. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while optical discs optically copy data using lasers. Combinations of the above should also be included within the scope of computer-readable media. Additionally, the operation of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transient processor-readable medium and / or computer-readable medium, and may be incorporated into a computer program product.

[0126] The prior description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be apparent to those skilled in the art, and the principles defined herein can be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Therefore, this disclosure is not intended to be limited to the embodiments shown herein, but is to be given the broadest scope consistent with the following claims and the principles and novel features disclosed herein.

[0127] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The disclosed aspects and embodiments are for illustrative purposes and are not intended to be limiting, wherein the true scope and spirit are indicated by the following claims.

Claims

1. A computer-implemented method for position prediction using an end-to-end target structure tracking system, comprising: A machine learning model is executed by a computer to extract a feature set from imaging projection data associated with target structures in relation to the patient's anatomy. The computer executes a reconstruction algorithm to transform the extracted feature set into a feature set in three-dimensional space; The computer executes a recurrent neural network to obtain three-dimensional feature map data associated with the target structure, the recurrent neural network being configured to sort the imaging projection data using the feature set in three-dimensional space; The computer extracts a template feature map from a three-dimensional simulation image, the template feature map including the target structure; The computer compares the template feature map with the three-dimensional feature map data; as well as The computer indicates the probability that the positions of three-dimensional points in the template feature map match those of three-dimensional points in the three-dimensional feature map data.

2. The computer-implemented method according to claim 1, further comprising: The computer receives segmentation information associated with the target structure; as well as The computer uses the segmentation information and the three-dimensional simulation image to extract the template feature map from the three-dimensional simulation image.

3. The computer-implemented method according to claim 1 or 2, further comprising: The computer executes a forward projection algorithm to transform the feature set in three-dimensional space into a feature set in two-dimensional space; as well as The computer feeds the feature set in the two-dimensional space into the machine learning model.

4. The computer-implemented method according to claim 1, 2 or 3, wherein the reconstruction algorithm is a second machine learning model, such that the end-to-end target structure tracking system is trained using a loss function that considers a probability distribution of peak-to-sidelobe ratio, the probability distribution corresponding to the probability of a 3D point in the template feature map matching the position of a 3D point in the 3D feature map data.

5. The computer-implemented method according to any one of claims 1 to 4, further comprising: The computer determines the deformation field associated with the target structure; as well as Using the deformation field, the computer indicates the probability that a 3D point in the template feature map matches the position of a 3D point in the 3D feature map data.

6. The computer-implemented method according to any one of claims 1 to 5, further comprising: The computer determines the confidence value based on the peak and sidelobes associated with a probability distribution corresponding to the probability that the positions of three-dimensional points in the template feature map match those of three-dimensional points in the three-dimensional feature map data.

7. The computer-implemented method according to any one of claims 1 to 6, further comprising: The computer determines the classification of points in the 3D feature map data based on the probability that the positions of 3D points in the template feature map match those of 3D points in the 3D feature map data meet a threshold.

8. The computer-implemented method according to any one of claims 1 to 7, wherein the imaging projection data is based on at least one of a stereoscopic projection pair or a projection set generated using a multi-view imaging system.

9. The computer-implemented method according to any one of claims 1 to 8, further comprising: The computer extracts additional template feature maps from the three-dimensional simulation image associated with the additional target structure; The computer compares the additional template feature map with the three-dimensional feature map data; as well as The computer generates a multi-channel probability distribution, which indicates the probability that a 3D point in the template feature map matches the position of a 3D point in the 3D feature map data, and the probability that a 3D point in the additional template feature map matches the position of a 3D point in the 3D feature map data.

10. The computer-implemented method according to any one of claims 1 to 9, wherein comparing the template feature map with the three-dimensional feature map data comprises: Each point of the template feature map is convolved with each point of the 3D feature map data.

11. An electronic system comprising: A server includes a processor and a non-transient computer-readable medium, the non-transient computer-readable medium including instructions that, when executed by the processor, cause the processor to perform operations, the operations including: Execute a machine learning model to extract a feature set from imaging projection data associated with target structures related to the patient's anatomy; A reconstruction algorithm is executed to transform the extracted feature set into a feature set in three-dimensional space; A recurrent neural network is executed to obtain three-dimensional feature map data associated with the target structure, the recurrent neural network being configured to sort the imaging projection data using the feature set in three-dimensional space; A template feature map is extracted from a three-dimensional simulation image, the template feature map including the target structure; Compare the template feature map with the 3D feature map data; and This indicates the probability that the position of a 3D point in the template feature map matches that of a 3D point in the 3D feature map data.

12. The system of claim 11, wherein the processor is further configured to perform operations including: Receive segmentation information associated with the target structure; as well as The template feature map is extracted from the three-dimensional simulation image using the segmentation information and the three-dimensional simulation image.

13. The system of claim 11 or 12, wherein the processor is further configured to perform operations including: Perform a forward projection algorithm to transform the feature set in three-dimensional space into a feature set in two-dimensional space; as well as The feature set in the two-dimensional space is fed into the machine learning model.

14. The system according to claim 11, 12 or 13, wherein the reconstruction algorithm is a second machine learning model.

15. The system according to any one of claims 11 to 14, wherein the processor is further configured to perform operations, the operations including: Determine the deformation field associated with the target structure; as well as The deformation field is used to indicate the probability that a 3D point in the template feature map matches the position of a 3D point in the 3D feature map data.

16. The system according to any one of claims 11 to 15, wherein the processor is further configured to perform operations, the operations including: The confidence value is determined based on the peak value and sidelobe associated with a probability distribution corresponding to the probability that the positions of three-dimensional points in the template feature map match those of three-dimensional points in the three-dimensional feature map data.

17. The system according to any one of claims 11 to 16, wherein the processor is further configured to perform operations, the operations including: Based on the probability that the positions of the three-dimensional points in the template feature map match the positions of the three-dimensional points in the three-dimensional feature map data meet a threshold, the classification of the points in the three-dimensional feature map data is determined.

18. The system according to any one of claims 11 to 17, wherein the imaging projection data is based on at least one of a stereoscopic projection pair or a projection set generated using a multi-view imaging system.

19. The system according to any one of claims 11 to 18, wherein the processor is further configured to perform operations including: Extract additional template feature maps from the three-dimensional simulation image associated with the additional target structure; The additional template feature map is compared with the three-dimensional feature map data; as well as A multi-channel probability distribution is generated, the multi-channel probability distribution indicating the probability that a 3D point in the template feature map matches the position of a 3D point in the 3D feature map data, and the probability that a 3D point in the additional template feature map matches the position of a 3D point in the 3D feature map data.

20. The system according to any one of claims 11 to 19, wherein comparing the template feature map with the three-dimensional feature map data comprises: Each point of the template feature map is convolved with each point of the 3D feature map data.

Images