Lesion association using adaptive search and system for implementing same

Abstract

Lesion association using adaptive search and a system for implementing the same. Disclosed herein is a system for correlating images of lesions taken over different time periods, comprising an imaging device operable to image one or more lesions present in a living body. The imaging device captures a first image at a first point in time T1 and a second image at a second point in time T2. The microprocessor is operable to receive a first image and a second image and perform an adaptive search on the respective images. The adaptive search includes: selecting a first voxel in a first lesion in the first image; and radially searching in the second image for one or more second lesions sharing one or more overlapping first voxels with the first lesion in the first image. A probability is assigned if there is an overlap between the first lesion and the one or more second lesions. Each voxel in the first lesion is labeled based on the probability.

Description

Background Art

[0001] The present disclosure relates to lesion linking using adaptive search. More specifically, the present disclosure relates to linking lesions imaged at different time points.

[0002] The growth of malignant lesions is not easily controlled. Thus, they can spread locally as well as to distant parts of the body. They spread via the bloodstream or the lymphatic system to produce new lesions. Each lesion within the same patient may respond differently to treatment or have a different growth pattern. Thus, lesion-level assessment is desirable for a comprehensive understanding of disease response and better treatment management. However, lesion-level assessment is particularly difficult, especially in cases with many corresponding lesions, and may require manual matching, which is a tedious, time-consuming, subjective, and error-prone task.

[0003] Medical imaging, such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), or single photon emission computed tomography (SPECT), is an important tool for identifying malignant lesions and monitoring their response to treatment.

[0004] The radiological interpretation of PET / CT scans mentioned for oncology may involve the segmentation of a large number of suspicious cancer findings. Longitudinal PET / CT scans are routinely performed to assess disease progression and / or response to therapy. While there are algorithms to segment lesions at individual time points, there is no method to robustly link lesions across time points.

[0005] Accordingly, it is desirable to develop methods by which lesions can be tracked based on their origin and spread. Summary of the Invention

[0006] Disclosed herein is a system for correlating images of lesions taken at different time periods, comprising: an imaging device operable to image one or more lesions present within a living organism; wherein the imaging device takes a first image at a first time point T 1 and takes a second image at a second time point T 2Take a second image at a location; a microprocessor operable to receive the first image and the second image and perform an adaptive search on the corresponding images; wherein the adaptive search includes: selecting a first voxel in a first lesion in the first image; radially searching in the second image for one or more second lesions that share one or more overlapping first voxels with the first lesion in the first image; wherein the radial search is continuously performed with a gradually increasing radius until a maximum radial threshold τ; and wherein: if no overlap is found when the threshold τ is reached, there is no correspondence between the first lesion and the one or more second lesions; or if an overlap is found between the first lesion and the one or more second lesions, a determination is made regarding the number of first voxels that can fit each overlapping volume between the first lesion and the one or more second lesions; and a probability is determined based on the ratio of the first voxels that fit each overlapping volume between the first lesion and each of the one or more second lesions; wherein the probability is calculated by dividing the number of first voxels present in each overlapping volume by the total number of first voxels in all overlapping volumes between the first lesion and the one or more second lesions; and each voxel in the first lesion is labeled based on the probability.

[0007] Also disclosed herein is a method for correlating lesions imaged at different time periods, including: at a first time point T 1 Take a first image of one or more lesions present in a living body; at a second time point T 2 Take a second image of the one or more lesions; perform an adaptive search on the first image and the second image; wherein the adaptive search includes: selecting a first voxel in a first lesion in the first image; radially searching in the second image for one or more second lesions that share one or more overlapping first voxels with the first lesion in the first image; wherein the radial search is continuously performed with a gradually increasing radius until a maximum radial threshold τ; and wherein: if no overlap is found when the threshold τ is reached, there is no correspondence between the first lesion and the one or more second lesions; or if an overlap is found between the first lesion and the one or more second lesions, a determination is made regarding the number of first voxels that can fit each overlapping volume between the first lesion and the one or more second lesions; and a probability is determined based on the ratio of the first voxels that fit each overlapping volume between the first lesion and each of the one or more second lesions; wherein the probability is calculated by dividing the number of first voxels present in each overlapping volume by the total number of first voxels in all overlapping volumes between the first lesion and the one or more second lesions; and each voxel in the first lesion is labeled based on the probability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic depiction of a method detailing the lesion matching process;

[0009] Figure 2A is a depiction of two images taken at times T 1 and T 2 to illustrate how adaptive search is used to match lesions;

[0010] Figure 2B is a depiction of an exemplary method for determining the overlap between voxels in a lesion imaged at times T 1 and T 2 ;

[0011] Figure 3 depicts a similarity map of all lesions present in images taken at times T 1 and T 2 ;

[0012] Figure 4 depicts a similarity map and lesion class assignment for a simulation example; and

[0013] Figure 5 depicts a lesion mapping process for multiple images obtained at different times from T 1 to T n . Detailed Description

[0014] Definition

[0015] An adaptive search algorithm is a meta - heuristic algorithm applied to combinatorial optimization problems. It consists of an iteration composed of the successive construction of randomized solutions and their subsequent iterative improvement by local search. The randomized solutions are generated by adding elements from a list of elements sorted by a greedy function according to the quality of the solution they will achieve. A greedy algorithm is a method for solving a problem by choosing the best available option at the current time. It does not itself care whether the current best result will lead to an overall optimal result.

[0016] The adaptive search algorithm calculates the overlap between voxels in one or more lesions imaged at time point 2 (T 2 ) and those voxels in the lesions imaged at time point 1 (T 1 ), and this overlap is used to calculate a probability, which is then used to assign a label to those specific voxels. Each lesion imaged at time point T 2 has a unique label, such as 1, 2, …, N. All voxels of the tumor imaged at T 1 are labeled based on this probability.

[0017] Lesion masking involves creating a layer that covers the lesion (and only the lesion). Lesions are typically masked manually. Lesion masking is sometimes referred to as lesion mapping or segmentation.

[0018] A voxel is a 3D cube that can be used to create 3D models. It is similar to a pixel in a 2D image, but it has an additional dimension (Z) that allows it to have depth. A voxel may have a specific position in a 3D grid and a color value (or characteristic property) assigned to it. By combining multiple voxels with different characteristic properties and positions, complex shapes and objects can be created. Examples of characteristic properties are geometric, functional, or structural features.

[0019] Image registration is a tool that facilitates the spatial mapping of voxels so that it can be used in an adaptive search to obtain an adaptive overlap that will be used in the lesion association process.

[0020] Image alignment refers to the process of overlaying or matching corresponding features of two or more images to bring them into a common reference frame. It is commonly used in image processing and computer vision tasks such as image stitching, object recognition, and 3D reconstruction.

[0021] Disclosed herein is a system and method for performing an adaptive search to track the development of N lesions across multiple consecutive time points T 1 、T 2 、T 3 、……、T n-1 、T n where T n >T n-1 、……, where T 3 >T 2 and T 2 >T 1 where N and n are both integers having a value of 1 or greater. The method accurately correlates lesions between consecutive longitudinal PET / CT image scans of a single patient over a period of time. In an embodiment, the adaptive search for N lesions is performed at different consecutive time intervals, and the information derived therefrom can be integrated with information derived from other sources such as CT and PET images, such as radiomics information (e.g., the texture of the lesion); geometric information (e.g., the size and shape of the lesion); quantitative information (e.g., an increase or decrease in the number of lesions); functional information (e.g., how the lesion responds to treatment); structural information (e.g., whether it affects nearby structures, whether it is a non-occupying lesion) or the like.

[0022] In an embodiment, the overlap obtained from the adaptive search can be layered with information related to mismatches between PSMA (prostate-specific membrane antigen) and FDG (fluorodeoxyglucose) for use in radionuclide therapy selection. This involves assessing the suitability of using specific radiotracers for targeted therapy in patients with prostate cancer. PSMA-targeted radionuclide therapy and FDG PET / CT are two different types of imaging and therapy methods used in the management of prostate cancer. PSMA-targeted radionuclide therapy utilizes radiolabeled small molecules that target PSMA, which is highly expressed on the surface of prostate cancer cells. This method allows for the selective delivery of radiation to cancer cells while sparing the surrounding healthy tissue. On the other hand, FDG PET / CT is a diagnostic imaging technique that uses a radiotracer to visualize the metabolic activity of cells and is commonly used for cancer staging and detection of metastases.

[0023] In an embodiment, the method includes adding additional layers related to image-based information and / or non-image-based information to the first and second images in a multi-tracer study to determine mismatches for radionuclide therapy selection. In an embodiment, among other things, the multi-tracer study includes the use of PSMA and FDG. Assessing the PSMA and FDG mismatch involves using FDG PET / CT to analyze the expression of PSMA and the metabolic activity of the tumor. This assessment helps determine whether PSMA-targeted radionuclide therapy or FDG PET / CT will be more suitable for a particular patient. The aim is to identify patients who are more likely to benefit from PSMA-targeted therapy based on the presence of PSMA-positive tumors and minimal FDG uptake, thus indicating a favorable PSMA-FDG mismatch.

[0024] By evaluating such a mismatch, clinicians can make more informed decisions regarding the selection of the most appropriate treatment modality for an individual patient. This personalized approach can lead to improved treatment outcomes and better management of prostate cancer, ultimately enhancing the quality of patient care and life.

[0025] The information obtained from the adaptive search disclosed herein can also be integrated with other non-image-based information derived from other techniques, such as liquid biopsy (which provides information for detecting the functional molecular characteristics of tumors or diseases through comprehensive and quantitative molecular analysis of a patient's liquid sample (primarily plasma) from circulating free tumor deoxyribonucleic acid (ctDNA) or circulating free tumor ribonucleic acid (ctRNA) or both (circulating free total nucleic acid, ctNA)). In an embodiment, the adaptive search results can be verified and validated by medical professionals to ensure the accuracy of the matched cancerous lesions.

[0026] A similarity matrix can be created that correlates the different behaviors (appearance, disappearance, merging, splitting, … or the like) of different lesions that will be created or disappear using the same or different treatments. The lesions from N time points are encoded using a customized data structure and the interactions between the lesions are associated using the N-1 similarity matrices obtained by matching each pair of the N time points.

[0027] The method can also be used in multi-tracer studies to evaluate the concordance or discordance of lesions. The concordance and discordance of lesions refer to the agreement or disagreement, typically in the context of medical imaging and diagnosis, between different diagnostic tests, imaging modalities, or observers in terms of the identification and characterization of lesions. Concordance of lesions means that multiple diagnostic tests or imaging modalities or different observers yield consistent results, thus indicating a high degree of agreement in terms of the identification and characterization of lesions. A high degree of agreement indicates that the findings are reliable and the lesions are accurately identified and classified.

[0028] In daily clinical practice, manually correlating lesions and lesion-level disease analysis is challenging and impractical, especially for challenging cases with a large number of lesions that merge or split over time. The challenges include identifying the anatomical spatial correspondence between consecutive PET / CT studies, where there may be significant differences in patient position or body habitus. Multiple scenarios need to be considered as lesions may have appeared, disappeared, changed shape, grown, shrunk, split into multiple parts, or merged with adjacent lesions.

[0029] Figure 1 is a schematic depiction of method 100 that details the tumor matching process. The method includes imaging a region of a patient containing one or more lesions at several different time points (T 1 、T 2 、T 3 、…、T n-1 、T n ) and establishing a spatial correspondence between the images of the lesions taken at different times. The spatial correspondence refers to the matching of corresponding points in two or more images. In step 302, a region containing a lesion is selected and a first image is obtained at time T 1 . In step 304, a second image is obtained at time T 2 (where T 2 is greater than T 1 ). The lesion masking for the image obtained at time T 1 can also be used at time T 2Images taken at. Positron emission tomography (PET) / computed tomography (CT), magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT), or the like or combinations thereof can be used to obtain all images (first image, second image, etc.). In an embodiment, a single imaging device (e.g., positron emission tomography / computed tomography) is used to perform at different times T 1 , T 2 , T 3 , ……, T n-1 , T n All images obtained at. At different times T 1 , T 2 , T 3 , ……, T n-1 , T n The single imaging devices used at can each be provided by different manufacturers. In an embodiment, a single imaging device can generate more than one imaging modality, such as for example PET and CT or MRI and CT. In other words, if the first image taken at time T 1 is obtained using positron emission tomography / computed tomography, then all subsequent images used in the image correlation process are also obtained using positron emission tomography / computed tomography. In this particular example, information from other images such as MRI, SPECT, liquid biopsy, etc. can be layered onto the images derived from positron emission tomography / computed tomography to generate a more accurate correlation between the images over time and also to more accurately classify lesions.

[0030] In step 306, spatial mapping can be performed to correlate one or more lesions in the first image taken at time T 1 with those lesions in the second image taken at time T 2 . Note that spatial mapping is not only performed between the images obtained at times T 1 and T 2 as elucidated in Figure 1 , but as will be detailed later, can be used to correlate lesions imaged at multiple times T 1 , T 2 , T 3 , ……, T n-1 , T n . Spatial mapping involves establishing a spatial correspondence between the images of one or more lesions at different times during the life cycle of the lesion. Spatial mapping is also used to provide at T 1The correspondence between the voxels of the first image taken at [location]. Over a period of time, a particular individual lesion may continue to grow in size, change shape, split into several lesions, merge with other lesions, or shrink and disappear due to treatment.

[0031] Now referring again to Figure 1 , there are several ways by which the lesions imaged at time T 1 can be associated (spatially mapped) with those lesions imaged at time T 2 . These are broadly classified in Figure 1 as step 308. In one embodiment, the spatial mapping can be performed via mask matching (detailed below). In another embodiment, the spatial mapping can be performed via image alignment (where the images obtained at different times T 1 , T 2 , T 3 , ……, T n-1 , T n are all arranged to have the same coordinates), and a sequential search is performed in 3D space with different sizes of radii (up to a maximum threshold radius τ) using the voxels in the lesion (the selected voxels are called "landmark voxels") as pivot points. The landmark voxels have the property being searched for. The landmark voxels can be located at any position within the lesions seen in the image taken at time T 1 .

[0032] In an embodiment, the landmark voxels are located at the centroid of at least one lesion imaged at time T 1 . This involves Figure 1 step 310 in 2 . A continuous search is performed with different sizes of radii until the threshold maximum radius τ, where the maximum threshold τ is greater than the maximum voxel size. If, during the search of the image at time T 1 , voxels with properties similar to the landmark voxels are detected in overlapping lesions, a determination is made regarding the number of such landmark voxels that can fit in the particular overlapping region. A probability determination is made as to whether it corresponds to a lesion seen in the previous image at time T Figure 1 (see steps 312 and 312A in 1 ). If no voxels with properties similar to the landmark voxels are detected, a continuous search is performed with continuously increasing sizes of radii until the maximum threshold radius τ is reached (see steps 314 and 316). If, despite reaching the maximum threshold radius τ, no voxels with properties similar to the landmark voxels are found, the landmark voxels imaged at time T 2 , T 3 , ……, T n-1 and T nNo correspondence is established between the subsequent images obtained. For the lesions detected in step 312A, imaging features for merging and splitting the lesions are calculated (see step 318), and a similarity matrix is developed that determines the lesion associations between the images taken at different times T 1 、T 2 、T 3 、……、T n-1 、T n (see step 320).

[0033] In an embodiment, spatial mapping can be performed by matching the mask used in the first image at time T 1 with the image obtained at time T 2 respectively. This corresponds to Figure 1 step 306 in

[0034] If desired, the images can be lesion masked. Masking is a technique used in image processing to selectively process certain regions of an image while ignoring others. In the context of lesion identification, masking can be used to highlight or isolate specific regions of an image that contain lesions, making them easier to identify and analyze. When generating a lesion mask, certain known features of a specific lesion obtained from a reference dataset and a simulated dataset can be utilized, such as, for example, total lesion volume, average lesion size, lesion count, etc. Mask modeling can also be performed to facilitate improved accuracy in lesion masking using regression modeling with maximum likelihood estimation, mean bias reduction estimation, or spatial Bayesian modeling, where the regression coefficients have a conditional autoregressive model to account for local spatial dependence. Generating an accurate lesion mask is useful in the spatial mapping step 304, which is detailed below. Segmentation software can be used to create an initial lesion mask.

[0034] Mask matching is a fast way to achieve the results obtained via steps 308, 310, 312, and 312A. In mask matching, a prior mask (e.g., a mask developed for the image obtained at time T 1 ) is used to exclude irrelevant background regions in the image obtained at time T 2 . When generating the mask, the co-occurrence region between the support image (the image obtained at T n-1 ) and the query image (the image obtained at T n ) is obtained, and this can be used as a prior mask to exclude irrelevant background regions. The query image is the image being analyzed to identify the presence of lesions. It is compared with the support image containing known lesions to determine whether the query image contains similar lesions. The results are then concatenated (i.e., associated together in a chain or series) and sent to an inference module (using appropriate software) to facilitate spatial mapping. Mask mapping is sometimes also referred to as template mapping.

[0035] Another method for establishing spatial correspondence between images taken at different times involves using a spatial mapping algorithm for image alignment. There are two main types of image alignment algorithms: intensity-based and feature-based. Intensity-based methods compare intensity patterns in images via a correlation metric, while feature-based methods find correspondences between image features such as points, lines, and contours. Spatial correspondence can be established by transforming one image to align with another. The reference frame in the target image is fixed, while other data sets are transformed to match the target. There are many algorithms available for image alignment and spatial correspondence, including linear transformation, radial basis functions, and large deformation models. Image similarity is widely used in medical imaging. Image similarity metrics quantify the degree of similarity between intensity patterns in two images. The choice of image similarity metric depends on the modality of the images to be registered. Common examples of image similarity metrics include cross-correlation, mutual information, sum of squared intensity differences, and ratio image uniformity. Mutual information and normalized mutual information are the most popular image similarity metrics for the registration of multimodal images. Cross-correlation, sum of squared intensity differences, and ratio image uniformity are typically used for the registration of images within the same modality.

[0036] As described above, an adaptive search is performed to correlate lesions over different time periods, and this includes Figure 1 steps 308, 310, 312, and 312A in 1 . The adaptive search can be performed using an adaptive search algorithm. The adaptive search algorithm can assign a given probability that each voxel of each lesion imaged at time point 1 (T 2 ) matches a voxel in all lesions imaged at time point 2 (T 2 ), where time point T 1 is different from time point T Figure 2A and 2B . This is described and discussed in

[0037] Figure 2A is a depiction of two images taken at times T 1 and T 2 to illustrate how an adaptive search can be used to achieve spatial mapping while considering and excluding potential errors. FIG. 2 depicts two images - a first image 1002 and a second image 1004 of patient 1000 taken at times T 1 and T 2 , respectively. All lesions seen in the examination images 1002 and 1004 are examined, and a one-to-one spatial correspondence or spatial mapping of each voxel in at least one lesion in both of these corresponding images is established to facilitate image alignment. The spatial correspondence of each voxel is indicated as T d . T dThe purpose is to enable the realization of T 1 the mapping from any voxel position in the image to its corresponding position in the T 2 image. The spatial correspondence T d is a term used to describe the process of finding the best match between two images taken at different times (T 1 and T 2 ) using a transformation matrix. The transformation matrix is a mathematical representation of how the images are related in terms of rotation, translation, scaling, and skew. The goal of the spatial correspondence T d is to align the images so that they can be compared or fused for various applications such as image alignment, image segmentation, image fusion, or image analysis. One way to perform the spatial correspondence T d is to use a feature-based method that extracts salient points or regions from the images and then finds the correspondences between them using descriptors or similarity metrics. Another way is to use an intensity-based method that directly compares the pixel values of the images and optimizes a cost function that measures their similarity or dissimilarity. Both methods use an initial guess or estimate of the transformation matrix, which can be refined iteratively until a satisfactory result is achieved.

[0038] Once the image alignment is established, an adaptive search process for the image acquired at time T 2 is initiated to determine whether other lesions present in the image taken at time T 2 have some correspondence with the lesions in the image taken at time T 1 . In an embodiment, the search can be performed via a scalar search. The scalar search includes using a marked voxel (also referred to as the first voxel) in the first lesion (in the first image 1002 imaged at T 1 ) as a pivot point with a radius, and performing a radial search in 3D for other voxels present in the image taken at time T 2 . In an embodiment, the search radius is centered at a point in the second lesion (in the second image 1004) that corresponds to the position of the first voxel in the first lesion (in the first image 1002). The initial radius used in the search is the size of one voxel (such as, for example, it can correspond to the size of the first voxel in the first image 1002 imaged at T 1 ). The radius only has magnitude (i.e., the initial radial size of one voxel), but no associated direction (i.e., characteristic property). If a second voxel (with the characteristics of the first voxel) is located in the second image 1004 at time T 2 during the scalar search, then regarding whether the found second voxel (in the second lesion in the second image) has the same characteristics as the first voxel at time T 1The characteristics corresponding to the labeled voxels (first voxels) of the first lesion imaged at [location] are analyzed a second time. If no such voxels are located, a successive search is performed with an increasing radius size until a threshold radius size τ is reached. If no voxels are found in all successive searches, at time T 1 There is no correspondence between the lesion imaged at [location] and the lesion in the image at time T 2 [location].

[0039] As described above, if voxels are found in the second image, they are analyzed for the characteristics of the labeled voxels in the first image. If the voxels imaged at T 2 [location] and the characteristics of the labeled voxels at T 1 [location] have a correspondence, then a determination is made that the voxels image at T 2 [location] (and thus the lesion in the second image 1004) does indeed correspond to the labeled voxels (and thus the lesion) from the first image 1002 taken at T 1 [location]. The probability determination includes determining the characteristics (of the second lesion) including geometry, structure, and / or function, and determining whether these characteristics correspond to those of the first lesion.

[0040] In summary, the scalar search involves two steps - the first step in the search process involves determining the presence of new lesions at time T 1 [location] in addition to the lesions present at time T 2 [location]. The second step in the process is to determine whether the new lesions found at time T 2 [location] correspond to the characteristics of any of the lesions seen in the image at time T 1 [location]. Any lesions found at time T 2 [location] (with the desired characteristics) are then spatially mapped to the lesions seen in the image at time T 1 [location]. In this way, associations can be established between multiple images taken at times T 1 T 2 T 3 …, T n-1 up to T n [location]. The associations between lesions established via multiple images taken at times T 1 T 2 T 3 …, T n-1 up to T n [location] can be achieved via a similarity matrix detailed below.

[0041] In another embodiment, after image alignment, spatial mapping can be performed using vector search. Vector searches may take less time than scalar searches because they may involve only a single step for the search. This is because the vectors involved in the radial search have attached to them both their magnitude (e.g., the initial radius of 1 voxel) and direction (e.g., characteristics of the labeled voxel such as lesion volume, lesion geometry, or the like). To use vector search, images obtained from an imaging device such as PET / CT can be attached with relevant characteristics based on geometry, structure, and / or function. Since the images have both magnitude and direction characteristics, they can undergo vector search. Vector search is a type of search that uses vectors to represent images. A vector is a mathematical representation of an image that can be used to compare images and find similarities between them. In vector search, images are represented as vectors and then compared with other vectors to find similar images. By comparing the distances and similarities between vectors, associations between several images (taken at different times) can be achieved.

[0042] In vector search, the initial search vector combines a magnitude component (e.g., the radius of the labeled voxel) with a direction component (e.g., one or more characteristics (such as geometric, functional, or structural) of the labeled voxel in the lesion imaged at time T 1 ). A search is made for other vectors in the image at time T 2 that have the characteristic attributes of the search vector. The voxel whose voxel radius and characteristic attributes are being searched is typically referred to as the "labeled voxel". It can be located anywhere in the lesion imaged at time T 1 . The vector search radius typically pivots around the labeled voxel located in the lesion imaged at time T 1 . In an embodiment, the labeled voxel is located at the centroid of the lesion imaged at time T 1 , and the vector search radius pivots around the center of the labeled voxel located at the centroid of the lesion. If no other identical vectors are identified in the initial search, the search vector radius (which is a combination of voxel size and characteristic attributes) is continuously increased until a maximum predefined threshold τ of the vector size is reached.

[0043] If during the search, it is determined that another vector in the image taken at time T 2 has characteristic attributes that are substantially similar to that vector (which has the additional size and characteristics of the labeled voxel), a probability determination is made as to whether that vector corresponds to the lesion seen in the image at time T 1 . In the case of vector search, a similarity matrix can also be created, and associations can be achieved between images taken at multiple times T 1 up to T n .

[0044] The different types of searches detailed above can be illustrated by Figure 2A and 2B the examples provided in Figure 2A and 2B The interpretation of image registration and corresponding mapping of lesions imaged at different times (using scalar search) will now be explained with reference to the images in Figure 2A The adaptive search process is detailed, where each voxel in the lesion imaged at T 1 is assigned a unique label (based on the highest probability), or no label is assigned if the search process does not locate any voxel that matches a voxel with a unique label. As detailed above, image registration is a tool that facilitates the spatial mapping of voxels such that it (the mapping) can be used in the adaptive search to obtain the adaptive overlap that will be used in the lesion association process.

[0045] As described above, Figure 2A depicts two images - the first image 1002 and the second image 1004 of patient 1000 taken at times T 1 and T 2 respectively. For example, referring to the image 1002 in Figure 2A , the lesion 1006A is imaged in a PET / CT scan taken at time T 1 . Although there are other lesions in the image 1002, only the lesion 1006A will be of concern for this discussion. The PET / CT scan image 1004 taken at time T 2 shows other lesions 1008A and 1008B in addition to the lesion 1006B. As a first step, if the voxels (all or part) in the lesion 1006A can only match the voxels in the lesion 1006B (they will have a one-to-one correspondence and no further analysis is performed).

[0046] The process is initiated by applying the spatial mapping (or transformation) obtained through image registration - or any other method - to each voxel in each lesion. After applying the transformation to a given voxel position in the lesion 1006A, it will be mapped to a position in T 2 and then the search process (as explained before) is started to obtain the probability that the labeled voxels belonging to the lesion 1006A will overlap with the voxels present in the lesion 1006B.

[0047] If the voxels (all or part) in the lesion 1006A only match the voxels in the lesion 1006B, it is a one-to-one correspondence and no further analysis of this lesion is required (other features are used later in the analysis and construction of the similarity matrix). Additional analysis is established for lesions that overlap with more than one lesion (based on the adaptive search).

[0048] Once image registration has been established, other new lesions near lesion 1006B can be searched for, and labeled voxels similar to those of lesion 1006A can be examined. In other words, lesions such as 1006B used in the image registration are excluded from the further search, and the further search is conducted to determine whether there are other lesions at time T 2 that have the voxel characteristics of lesion 1006A.

[0049] If another lesion overlaps with lesion 1006B from image 1004 taken at T 2 and the lesion (which overlaps with lesion 1006B) contains one or more labeled voxels identical to those of 1006A from image 1002 (taken at T 1 ), then the probability for that voxel is calculated based on the sum of the overlapping voxels seen in all overlapping lesions (which overlap with lesion 1006B). This method is detailed in Figure 2B . Figure 2B Details how lesions imaged at time T 1 can be associated with other overlapping lesions imaged at time T 2 .

[0050] As with Figure 2A , Figure 2B depicts two images with lesions (one obtained at T 1 and the other at T 2 ) and a spatially corresponding T d estimated as an initial step (which, as detailed above, can be obtained using image registration, point matching, or any other technique). Now referring to Figure 2B , then for all overlapping lesions imaged at T 2 that overlap with the lesion imaged at T 1 , the total number of overlapping voxels between the lesions imaged at T 1 is calculated. The probability for each of the voxels seen in the different overlapping lesions is calculated by ratioing the voxels seen in a particular overlapping lesion to the total number of overlapping voxels. Then these probabilities are used to assign the overlap between the lesions imaged at T 1 and those imaged at T 2 . (See steps 312 and 312A in Figure 1 ) This method can be repeated continuously for each subsequent image obtained at times T 3 , T 4 , ……, T n-1 , T n . In other words, at time T 3The image taken at [T] can (again by examining the overlapping voxels) be correlated with the image taken at T 2 The image taken at [T] can be correlated with the image taken at T 4 The image taken at [T] can be correlated with the image taken at T 3 and so on until the moment when the image taken at time T n can be correlated with the image taken at time T n-1 The overlapping volumes (based on the number of voxels of each lesion) are used in the construction of one feature in the similarity matrix (detailed below). Further analysis is performed to detail whether the lesions are splitting, merging, disappearing, etc., as detailed below.

[0051] For example, in Figure 2B the search space is the area shown within the dashed circle, which has the number of voxels found for each marker. In the image at time T 1 lesion L 0 is seen, while at time T2, in addition to the original lesion L 1 seen at T 0 there are four more lesions L 1 、L 2 、L 3 and L 4 . Each lesion imaged at time T 2 is assigned an identity (e.g., L 1 、L 2 、L 3 and L 4 ). After registering the lesion L d imaged at time T 2 (shown within the dashed circle) to the lesion L 0 at time T 1 via spatial correspondence T 0 , the number of overlapping voxels between lesion L 0 and lesions L 1 、L 2 、L 3 and L 4 is examined. As can be seen from Figure 2B there are two overlapping voxels between L 0 and L 1 , five overlapping voxels between L 0 and L 3 , and one overlapping voxel between L 0 and L 2 . There are no overlapping voxels between L 0 and L 4 .

[0052] The voxel (v) at time T 2The probability that a voxel belongs to each of the three labels can be calculated as follows (the number of voxels of each label / the total number of voxels):

[0053] P(v = L 1 ) = 2 / (1 + 2 + 5) = 0.25

[0054] P(v = L 2 ) = 1 / (1 + 2 + 5) = 0.125

[0055] P(v = L 3 ) = 5 / (1 + 2 + 5) = 0.625

[0056] P(v = L 4 ) = 0 / (1 + 2 + 5) = 0.0

[0057] Based on the probability of overlap, the similarity or association between lesions can be calculated. After repeating the above process for all voxels in lesion L 0 , the final result will be as shown in the figure of the image on the right side of the arrow, which represents the overlap between L 0 and other lesions in T 2 . In other words, each voxel in lesion L 2 is labeled based on the probability obtained from the number of overlapping voxels of each lesion (L 1 , L 2 , L 3 and L 4 ) seen in the image at T 0 . The overlapping volume (based on the voxel contribution of each label) is used in the construction and further analysis of a feature in the similarity matrix to obtain splitting, merging, and the like.

[0058] The search process detailed above is performed for each voxel within each lesion. The above adaptive search can be performed sequentially or simultaneously to associate multiple lesions seen in the image taken at time T 2 with lesions seen in the image taken at time T 1 . In an embodiment, multiple different first lesions (each of the first lesions having labeled voxels) imaged at time T 1 (on the image taken at time T n ) can be searched multiple times simultaneously or sequentially, and the first lesions overlap with multiple overlapping second lesions imaged at time T 2 , and each of the second lesions has the same labeled voxels in the overlapping region.

[0059] In an embodiment, each voxel within each lesion is searched. In other words, if at time T n-1There are N different lesions in the image at [location], and each lesion has m different voxel characteristics. For the purpose of spatial mapping, m(N - 1) different searches (each search having the same or different labeled voxels) can be performed sequentially or simultaneously on any subsequent image T n .

[0060] After completing the adaptive search for each voxel and assigning a unique label to that voxel, a similarity matrix is generated. Based on this, the overlap between lesions can be calculated (which is a characteristic of the similarity matrix). Other components of the similarity matrix can be imaging features (e.g., structural features, functional features, geometric features, and the like). In this way, a similarity matrix can be generated between multiple lesions present in the image taken at time T n and multiple lesions observed in the image taken at time T n-1 . In another embodiment, N - 1 adaptive searches can be performed simultaneously or sequentially on the different lesions present in the Nth image taken at time T n ; where both N and n are integers greater than or equal to 1. In an embodiment, n is 2 or greater.

[0061] Figure 3 Depicts an exemplary embodiment of a method for forming a similarity map or matrix. A similarity map is a visualization strategy that helps represent the similarity (and other characteristics) of lesions imaged at different times in an image. It can be used to visualize the origin of lesion similarity between different lesions imaged at different times. In geometry, two objects are similar if they have the same shape, or if one has the same shape as the mirror image of the other. More precisely, one object can be obtained from the other by uniform scaling (also known as a weighting factor), possibly along with additional translation, rotation, and reflection. In other words, if two objects are similar, each object is consistent with the result of a specific uniform scaling (weighting) of the other object. This means that either object can be rescaled, repositioned, and reflected so as to exactly coincide with the other object. Figure 3 Contains images of patient 1000 taken at time T 1 via CT (2002), PET (2004), and mask (2006), and again at time T 2 via CT (3006), PET (3004), and mask (3002). After image registration and spatial mapping (or spatial correspondence) via the adaptive search algorithm as described above, these images together with geometric information can be integrated and presented as a weighted mixture (also known as a similarity score S), as shown in formula (1)

[0062] S = WG x FG + Wp x Fp + Wc x Fc (1),

[0063] where F G , F P and F C represent vectors of geometric, functional, and structure-based features, while W G , W P and W c represent their respective weighting factors. As can be seen from Equation (1), each specific feature obtained from the image is weighted for each contribution mode. In Equation (1), each of the geometric (F G ), functional (F P ), and structure-based vectors (F C ) may depend on one or more factors. For example, the structure-based vector F C can be mathematically calculated from a number of different features as represented below in Equation (2).

[0064] where f 1 , f 2 , f 3 , f 4 , …, f n are the different weighted features considered in determining the structure-based vector F C .

[0065] A similarity score can be calculated for all lesions present in each image taken at different times. Figure 3 Depicts a similarity map 4000 of all lesions present in the images taken at times T 1 and T 2 . The similarity map represents a matrix (m, n), where m and n represent the number of lesions in the images taken at time points T 1 and T 2 . The x-axis represents the similarity scores of all lesions imaged at time T 2 , and extends from L 0 (the first lesion at each time point) to L n-1 (for n lesions) present in the image. The y-axis represents the similarity scores of all lesions imaged at time T 1 , and extends from L m-1 (for m lesions) present in the image to L 0 (the first lesion). Figure 4 The size of the similarity matrix in

[0066] is set to the maximum of (m, n) to handle the case where there are zero lesions at one time point. In this case, all imaged lesions will be new lesions. Figure 4 is explained below with reference toFigure 3 Explanation of overlapping similarities in the similarity mapping 4000 in Figure 4 depicts the similarity matrix of the simulation exercise, where at time T 1 the first image containing 11 lesions is mapped to the second image containing 17 lesions at time T 2 . Note that Figure 4 the specific similarity matrix in 1 and time T 2 shows the volume of the overlapping part between the lesions imaged at (which contain labeled voxels). The y-axis (vertical axis) represents the metabolic tumor volume (MTV) of the lesions imaged at time T 1 . The x-axis (horizontal axis) represents the metabolic tumor volume (MTV) of the lesions imaged at time T 2 . Different similarity matrices can be constructed to show other regions (e.g., geometric, functional, or structural features) of the overlapping features of the lesions imaged at time T 1 and time T 2 . Most of the numbers visible in the mapping are -1, which is the default value and reflects that there is no overlap between the lesions imaged at time T 1 and T 2 . Figure 4 The values in 1 and time T 2 indicate the overlap in terms of volume between the lesions imaged at (as mentioned above, similarity can also be for other features such as geometric, functional, or structural features). For example, if the lesion imaged at time T 1 has a volume of 100 cubic millimeters and 50% of this volume overlaps with an adjacent lesion when imaged at time T 2 (and the overlapping volume contains labeled voxels), then Figure 4 the overlap value in the mapping in

[0067] After creating the mapping, each row and column is systematically examined for overlapping similarities by looking at the values (other than -1) in any specific row and column and the number of these different overlap values (i.e., identifying matching scenarios). Referring to Figure 4 , the information (overlapping volume) of all the lesions on the y-axis comes from the previous image (taken at T n-1 ), where all the lesions imaged at T n-1 are represented in the rows of the table, and the overlapping volume of all the lesions on the x-axis comes from the current image (T n ), and is represented in the columns of the table. Based on this, it is possible to determine whether the lesions come from the previous image (taken at T n-1 ) (given row), compared to those from the current image (at T nThe lesions in the images taken at [time] do not overlap, overlap with one or more lesions (columns with any value other than -1).

[0068] If a particular row and column each have only one value (other than -1), it reflects a one-to-one mapping between the lesions in the two images. For example, the row represented by the number 5002 has only one value (which is circled). Similarly, the column 5010 that intersects row 5002 has only one value (which is the same circled value as at row 5002). This intersecting value indicates that this particular lesion has a one-to-one correspondence between the images taken at time T 1 and T 2 The special lesion may be shrinking or growing.

[0069] Rows with no values (other than -1) (see the row represented by the number 5004) indicate no overlap in terms of similarity (geometric, functional, or structure-based similarity) and the lesions are disappearing lesions. If there are two or more values (other than -1) in a row (see the row represented by the number 5006), it indicates that the lesion is a splitting lesion. Columns with no values (other than -1) indicate no overlap in terms of similarity (geometric, functional, or structure-based similarity) with the lesions at time T 1 and thus the lesions are new lesions (see the column with the number 5008). If a column has multiple values (e.g., two or more values) (other than -1), it means that two or more lesions are from a previous image (at time T 1 and are merged to form a lesion from the current image (at time T 2 ), which indicates a merged lesion. For example, column 5012 shows 3 values (other than -1). This means that three lesions from the previous image are merged into a lesion from the current image, which indicates a merged lesion. In other words, all the lesions from the previous image (indicated by the rows with values) are merged to form the lesions from the current image (identified by the columns being studied that intersect these rows with values other than -1).

[0070] Although Figures 1 - 4 most of the above discussion only relates to the images taken at two times T 1 and T 2 it is possible to correlate multiple images taken at different times T n . Figure 5 Method 6000 for achieving this is depicted. The method includes several steps. In step 6002, for each pair of time points T 1 and T 2 , T 2 and T 3 , T 3 and T4, and so on until T n-1 and Tn The images obtained at [location] undergo image alignment and are then mapped together spatially via the adaptive search mechanism detailed above (especially with reference to Figures 2 and 3).

[0071] In step 6004, using N - 1 similarity matrices, the customized graphical structure is iteratively updated. Nodes can represent lesions, and edges can represent different events. In step 6006, the spatial correspondences between each pair of time points in the similarity matrix are used in the adaptive search algorithm to estimate the overlap between lesions imaged at any time point and any earlier time point (as detailed in Figure 3 and 4 ). This is called recursive mapping. In step 6008, lesions that may disappear and reappear at a later time are examined, and the similarity matrix is updated. In step 6010, the generated graphical structure is traversed (as detailed in reference Figure 5 ) to generate the final association or mapping between lesions at time point T 1 and time point T n .

[0072] In an embodiment, the system for associating images can include an imaging device and a microprocessor (not shown). The imaging device includes at least one of a PET / CT / MRI / SPECT machine, which can image one or more lesions present in a living organism at different time points. The microprocessor is operable to perform an adaptive search algorithm on consecutive images obtained from the imaging device at different time points. The adaptive search algorithm can assign a given probability to each voxel of each lesion imaged at time point 1 (T 1 ), and this probability is used to match the voxel with voxels in one or more lesions imaged at time point 2 (T 2 ), where time point T 2 is different from time point T 1 . Other characteristic information (e.g., geometric, structural, and / or functional features) can be manually provided to the microprocessor for integration with the images. In another embodiment, this other characteristic information can be calculated by image analysis techniques and processed through constructing similarity matrices and further analysis. Alternatively, the characteristic information can be provided to the microprocessor via a data table stored in the microprocessor. In an embodiment, the microprocessor can integrate geometric, structural, and / or functional information with the images obtained from PET / CT / MRI / SPECT to create a synthetic image that can be used for vector search. In yet another embodiment, the geometric, structural, and / or functional information can be obtained directly from PET / CT / MRI / SPECT.

[0073] In an embodiment, the microprocessor can obtain from the PET / CT / MRI / SPECT machine at each pair of time points T1 and T 2 、T 2 and T 3 、T 3 and T4 and so on up to T n-1 and T n obtained images, perform image alignment, and then map the images together spatially via the adaptive search mechanism detailed above. The microprocessor is also operable to create N-1 similarity matrices and iteratively update the customized graphical structure to generate associations between lesions imaged from time point T 1 to T n The human operator can manually select which other feature information (geometric, structural, and / or functional features) to provide to the microprocessor to make a determination regarding the classification of lesions undergoing splitting, merging, shrinking, etc. In another embodiment, the microprocessor itself can select the feature information that enables it to make a determination regarding the classification of lesions undergoing splitting, merging, shrinking, etc. Lesion classification can be used by a doctor to determine the course of action or therapy for treating a patient.

[0074] Compared to other methods of lesion imaging, the methods and systems disclosed herein have many advantages. In addition to including other image features (texture or others), the introduction of the adaptive search mechanism also provides the ability to match lesions in several images taken at different times. This allows the calculation of the final similarity score for each tumor imaged at time T n-1 to be matched with another lesion imaged at T n where n is an integer value. Additionally, based on the associations and modeling of lesions imaged at different time points, it provides the ability to facilitate the assignment of lesions to one of different categories (splitting lesions, merging lesions, shrinking lesions, disappearing lesions, and the like). By including imaging information and geometric information in an adaptive manner, the method is more robust compared to other available methods.

[0075] The integration of other imaging features (e.g., geometric, functional, and / or structural features) can provide additional information to determine the similarity between lesions at different time points. Additionally, the ambiguities that may arise during the classification of lesions as a result of splitting (one-to-many) or merging (many-to-one) can be resolved by the integration of additional imaging features based on geometric, functional, and / or structural information.

[0076] In addition, the advantage of adaptive search is that it ensures that any errors that may be generated by the movement of certain parts of the body will be taken into account. In addition, in some cases, establishing spatial correspondence can be challenging. For example, compared to other time points, the patient's weight increases or decreases at one time point, or the lesion changes over time, such as decreasing or increasing in size. All of these problems can be addressed by adaptive search at each lesion level.

Claims

1. A system for correlating images of a lesion taken at different time periods, comprising: An imaging device operable to image one or more lesions present in a living body; wherein the imaging device captures a first image at a first time point T1 and captures a second image at a second time point T2; A microprocessor operable to receive a first image and a second image and perform an adaptive search on the respective images; wherein the adaptive search comprises: selecting a first voxel in a first lesion in a first image; radially searching in the second image for one or more second lesions that share one or more overlapping first voxels with the first lesion in the first image; wherein the radial search is performed continuously using gradually increasing radii up to a maximum radial threshold τ; and wherein: a) if no overlap is found when the threshold τ is reached, then there is no correspondence between the first lesion and one or more second lesions; or b) if an overlap between the first lesion and the one or more second lesions is found, making a determination as to the number of first voxels that can fit in each overlap volume between the first lesion and the one or more second lesions; and c) determining a probability based on a ratio of first voxels that fit into each overlapping volume between the first lesion and each of the one or more second lesions; wherein the probability is calculated by dividing the number of first voxels present in each overlapping volume by the total number of first voxels in all overlapping volumes between the first lesion and the one or more second lesions; and d) Labeling each voxel in the first lesion based on the probability. 2 . The system of claim 1 , wherein the initial radial search uses a search radius corresponding to the size of the first voxel. 3 . The system of claim 2 , wherein the search radius is centered on a point in one or more second lesions that corresponds to a location of a first voxel in a first lesion. 4 . The system of claim 1 , wherein the determining a probability further comprises including characteristics including geometry, structure, and / or function of the first tumor.

5. The system according to claim 1, further comprising: n Performing N-1 adaptive searches simultaneously or sequentially for different lesions present in the Nth image taken at the location; Where N is an integer of 2 or greater, and where n is a specific point in time from 1 to n-1; Where n is an integer.

6. The system according to claim 1, further comprising integrating the overlap and geometric information obtained from the first image and the second image to obtain a similarity score S of the lesion, as shown in formula (1): S=W G ×F G +W P ×F P +W C ×F C (1), where F G 、F P and F C represents vectors of geometric, functional, and structure-based features, and W G , W P and W c Denote their corresponding weighting factors.

7. The system of claim 6, further comprising creating a similarity table in which the overlap between lesions imaged at T1 is compared volumetrically with the overlap with one or more lesions imaged at T2; wherein all lesions imaged at time T1 are represented in rows of the similarity table, and wherein the one or more lesions imaged at time T2 are represented in columns of the table. The system of claim 7 , wherein the comparison of similarities is represented graphically.

9. The system of claim 8, wherein the graphical representation is arranged in rows and columns; and wherein each row or column can be evaluated for lesion characteristics including shrinkage, growth, division, or merging of a particular lesion.

10. The system of claim 1, wherein the adaptive search is facilitated by an adaptive search algorithm stored on the microprocessor.

11. The system of claim 9, wherein the lesion characteristics are used by a physician to prescribe therapy for a patient.

12. The system of claim 1, wherein the imaging device is a positron emission tomography (PET) scanner, an x-ray computed tomography (CT) scanner, a magnetic resonance imaging (MRI) scanner, a single photon emission computed tomography (SPECT) scanner, or a combination thereof.

13. A method for correlating lesions imaged at different time periods, comprising: Taking a first image of one or more lesions present in a living body at a first time point T1; capturing a second image of the one or more lesions at a second time point T2; performing an adaptive search on the first image and the second image; in Adaptive search includes: selecting a first voxel in a first lesion in a first image; radially searching in the second image for one or more second lesions that share one or more overlapping first voxels with the first lesion in the first image; wherein the radial search is performed continuously using gradually increasing radii up to a maximum radial threshold τ; and wherein: a) if no overlap is found when the threshold τ is reached, then there is no correspondence between the first lesion and one or more second lesions; or b) if an overlap between the first lesion and the one or more second lesions is found, making a determination as to the number of first voxels that can fit in each overlap volume between the first lesion and the one or more second lesions; and c) determining a probability based on a ratio of first voxels that fit into each overlapping volume between the first lesion and each of the one or more second lesions; wherein the probability is calculated by dividing the number of first voxels present in each overlapping volume by the total number of first voxels in all overlapping volumes between the first lesion and the one or more second lesions; and d) Labeling each voxel in the first lesion based on the probability.

14. The method of claim 13, wherein an initial search of the radial search uses a search radius corresponding to a size of the first voxel, and wherein the search radius is centered at a point in the second lesion corresponding to a location of the first voxel in the first lesion.

15. The method according to claim 13, further comprising: n Performing N-1 adaptive searches simultaneously or sequentially for different lesions present in the Nth image taken at the location; Where N is an integer of 2 or greater, and where n is a specific point in time from 1 to n-1; Where n is an integer.

16. The method according to claim 13, further comprising integrating the overlap and geometric information obtained from the first image and the second image to obtain a similarity score S of the lesion, as shown in formula (1): S=W G ×F G +W P ×F P +W C ×F C (1), Among them, F G 、F P and F C represents vectors of geometric, functional, and structure-based features, and W G , W P and W c Denote their corresponding weighting factors.

17. The method of claim 16, further comprising creating a similarity table in which the overlap between lesions imaged at T1 is compared volumetrically with the overlap with one or more lesions imaged at T2; wherein all lesions imaged at time T1 are represented in rows of the similarity table, and wherein the one or more lesions imaged at time T2 are represented in one or more columns of the table.

18. The method of claim 17, wherein the graphical representation is arranged in rows and columns; and wherein each row or column can be evaluated for lesion characteristics including shrinkage, growth, division, or merging of a particular lesion.

19. The method of claim 13, wherein the adaptive searching is facilitated by an adaptive searching algorithm stored on a microprocessor.

20. The method of claim 13, further comprising adding additional layers relating to image-based information and / or non-image-based information to the first and second images to generate a similarity matrix; wherein the image-based information is radiomic information, geometric information, quantitative information, functional information, structural information, or a combination thereof; and wherein the non-image-based information is derived from a liquid biopsy, comprising circulating free tumor deoxyribonucleic acid (ctDNA), circulating free tumor ribonucleic acid (ctRNA), or both (circulating free total tumor nucleic acid, ctNA).

21. The method of claim 20, further comprising adding additional layers of image-based information and / or non-image-based information to the first and second images in a multi-tracer study to determine mismatches in radionuclide therapy selection.

Images