System and method for one-time anatomical localization with unsupervised visual transducer for three-dimensional (3D) medical images

Abstract

A method for performing one-time anatomical positioning includes obtaining a medical image of a subject. The method includes receiving a selection of both a template image and a region of interest within the template image, wherein the template image includes one or more anatomical landmarks assigned with respective anatomical tags. The method includes inputting both the medical image and the template image into a trained visual transducer model. The method includes outputting both tile-level features and image-level features of both the medical image and the template image from a trained visual transducer model. The method further includes interpolating pixel-level features from the tile-level features of both the medical image and the template image. The method includes utilizing pixel-level features within a region of interest of a template image to locate and label corresponding pixel-level features in a medical image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application is a continuation-in-part of U.S. patent application serial number 18 / 475,406, filed on September 27, 2023, entitled "SYSTEM AND METHOD FOR ONE-SHOTANATOMY LOCALIZATION WITH UNSUPERVISED VISION TRANSFORMERS FOR THREE-DIMENSIONAL (3D) MEDICAL IMAGES," which is incorporated herein. Background Art

[0003] The subject matter disclosed herein relates to medical imaging, and more particularly, to systems and methods for one-shot anatomical structure localization utilizing an unsupervised visual transformer for three-dimensional (3D) medical images.

[0004] Non-invasive imaging techniques allow images of internal structures or features of a patient / object to be obtained without performing an invasive procedure on the patient / object. Specifically, such non-invasive imaging techniques rely on various physical principles (such as differential transmission of X-rays through a target volume, reflection of acoustic waves within a volume, paramagnetism of different tissues and materials within a volume, decomposition of target radionuclides within the body, etc.) to acquire data and construct an image or otherwise represent the observed internal features of the patient / object.

[0005] During MRI, when a substance such as human tissue is subjected to a uniform magnetic field (polarization field B0), the individual magnetic moments of the spins in the tissue attempt to align with the polarization field, but precess around it in a random order at their characteristic Larmor frequency. If the substance or tissue is subjected to a magnetic field (excitation field B1) that is in the xy plane and close to the Larmor frequency, the net alignment moment or "longitudinal magnetization" M z can be rotated or "tilted" into the xy plane to produce a net transverse magnetic moment M t After the excitation signal B1 is terminated, a signal is emitted by the excited spins and can be received and processed to form an image.

[0006] When these signals are used to generate images, the magnetic field gradient (G x , G y and G z). Typically, the area to be imaged is scanned in a series of measurement cycles in which these gradient fields are varied according to the particular localization method used. The resulting set of received nuclear magnetic resonance (NMR) signals is digitized and processed to reconstruct an image using one of the well-known reconstruction techniques.

[0007] For many medical imaging applications, including MRI, anatomical localization and landmark localization are important preprocessing steps to achieve the final application. For example, for MRI (such as AIR from General Electric Healthcare) x TM This is important for the development of smart prescription modules such as , automated image alignment in computed tomography (CT) imaging, or image contour driven registration. Typically, this process is currently done manually or through fine-tuning through various segmentation algorithms such as level sets or anatomical structure-based point landmark detectors. Each of these is specific to the anatomy of interest and must be repeatedly implemented based on changes in the anatomy or imaging data. This also includes generating ground truth for supervised segmentation (e.g., as with AIR x TM as achieved).

[0008] Furthermore, in clinical practice, patients are often followed at multiple time points using imaging as part of screening, lesion detection, treatment planning / management, and outcome studies. In such scenarios, clinicians repeatedly identify one or more lesions of interest at any given time point and then calculate disease velocity or progression using individual lesion metrics (e.g., maximum diameter, volume, shape, etc.). Summary of the invention

[0009] The following shows an overview of certain embodiments disclosed herein. It should be understood that these aspects are provided only to provide the reader with a brief overview of these specific embodiments, and these aspects are not intended to limit the scope of the present disclosure. In fact, the present disclosure may cover various aspects that may not be shown below.

[0010] In one embodiment, a computer-implemented method for performing one-time anatomical structure localization is provided. The computer-implemented method includes obtaining a medical image of a subject at a processor. The computer-implemented method also includes receiving a selection of both a template image and a region of interest within the template image at the processor, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels. The computer-implemented method also includes inputting both the medical image and the template image into a trained visual transformer model via the processor. The computer-implemented method even includes outputting both the block-level feature portions and the image-level feature portions of both the medical image and the template image from the trained visual transformer model via the processor. The computer-implemented method also includes interpolating the pixel-level feature portions from the block-level feature portions of both the medical image and the template image via the processor. The computer-implemented method further includes locating and labeling the corresponding pixel-level feature portions in the medical image using the pixel-level feature portions within the region of interest of the template image via the processor.

[0011] In another embodiment, a computer-implemented method for performing one-time anatomical structure positioning is provided. The computer-implemented method includes obtaining a medical image of a subject at a processor. The computer-implemented method also includes receiving a selection of a template image at the processor, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels, and a first reference point is marked on the template image. The computer-implemented method also includes inputting both the medical image and the template image into a trained visual transformer model via the processor. The computer-implemented method even includes outputting both the block-level feature parts and the image-level feature parts of both the medical image and the template image from the trained visual transformer model via the processor. The computer-implemented method also includes interpolating the pixel-level feature parts of the block-level feature parts from both the medical image and the template image via the processor. The computer-implemented method also includes clustering the pixel-level feature parts of both the medical image and the template image into anatomically similar regions via the processor. The computer-implemented method also includes assigning clustering labels to pixels of both the medical image and the template image in corresponding anatomically similar regions via the processor.

[0012] In another embodiment, a system for performing disposable anatomical structure positioning is provided. The system includes a memory that encodes a processor executable routine. The system also includes a processor that is configured to access the memory and execute the processor executable routine, wherein the routine causes the processor to perform an action when executed by the processor. The action includes obtaining a medical image of the subject. The action also includes receiving a selection of a template image, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels, and a first reference point is marked on the template image. The action also includes inputting both the medical image and the template image into a trained visual transformer model. The action even includes outputting both the block-level feature parts and the image-level feature parts of both the medical image and the template image from the trained visual transformer model. The action also includes interpolating the pixel-level feature parts of the block-level feature parts from both the medical image and the template image. The action also includes clustering the pixel-level feature parts of both the medical image and the template image into anatomically similar regions. The action also includes assigning clustering labels to pixels of both the medical image and the template image in corresponding anatomically similar regions.

[0013] In another embodiment, a computer-implemented method for performing concurrent longitudinal lesion detection and analysis is provided. The computer-implemented method includes obtaining medical imaging data of a subject at a processor, the medical imaging data being acquired at a time point later than when reference medical imaging data of the subject is acquired. The computer-implemented method also includes automatically marking, via the processor, one or more non-lesion areas and one or more lesions in the medical imaging data based on one or more non-lesion areas and one or more lesions marked in the reference medical imaging data. The computer-implemented method also includes inputting the marked medical imaging data into a trained visual transformer model via the processor. The computer-implemented method even includes outputting, via the processor, visual transformer features from the trained visual transformer model based on the marked one or more non-lesion areas and one or more lesion areas from the medical imaging data. The computer-implemented method further includes identifying, via the processor, similar feature voxels in the medical imaging data to identify one or more lesions based on the output visual transformer features. The computer-implemented method also includes labeling, via the processor, one or more lesions in the medical imaging data using a segmentation mask.

[0014] In even another embodiment, a system for performing concurrent longitudinal lesion detection and analysis is provided. The system includes a memory that encodes a processor executable routine. The system also includes a processor that is configured to access the memory and execute a processor executable routine, wherein the routine causes the processor to perform an action when executed by the processor. The action includes obtaining medical imaging data of a subject, which is collected at a time point later than the reference medical imaging data of the subject. The action also includes automatically marking one or more non-lesion areas and one or more lesions in the medical imaging data based on one or more non-lesion areas and one or more lesions marked in the reference medical imaging data. The action also includes inputting the marked medical imaging data into a trained visual transformer model. The action even includes outputting a visual transformer feature from the trained visual transformer model based on one or more non-lesion areas and one or more lesion areas marked from the medical imaging data. The action also includes identifying similar feature voxels in the medical imaging data to identify one or more lesions based on the outputted visual transformer feature. The action also includes labeling one or more lesions in the medical imaging data using a segmentation mask.

[0015] In yet another embodiment, a non-transitory computer-readable medium includes a processor executable code that causes the processor to perform an action when executed by the processor. The action includes obtaining medical imaging data of a subject, which is collected at a time point later than the time point of collecting reference medical imaging data of the subject. The action also includes automatically marking one or more non-lesion areas and one or more lesions in subsequent medical imaging data based on one or more non-lesion areas and one or more lesions marked in the reference medical imaging data. The action also includes inputting the marked medical imaging data into a trained visual transformer model. The action even includes outputting a visual transformer feature from the trained visual transformer model based on one or more non-lesion areas and one or more lesion areas marked from the medical imaging data. The action also includes identifying similar feature voxels in the medical imaging data to identify one or more lesions based on the output visual transformer feature. The action also includes labeling one or more lesions in the medical imaging data using a segmentation mask. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] These and other features, aspects, and advantages of the present subject matter will be better understood when the following detailed description is read with reference to the accompanying drawings, in which like characters refer to like parts throughout the several views, and in which:

[0017] Figure 1An embodiment of a magnetic resonance imaging (MRI) system suitable for use with the disclosed technology is illustrated;

[0018] Figure 2 A schematic diagram showing a visual transformer model according to aspects of the present disclosure;

[0019] Figure 3 A schematic diagram illustrating self-supervised training of a visual transformer model according to aspects of the present disclosure;

[0020] Figure 4 A flowchart showing a method for performing one-time anatomical structure localization (eg, when the anatomical structure is clear in an image (eg, a template image)) according to aspects of the present disclosure;

[0021] Figure 5 A schematic diagram showing querying a region of interest using tiles (eg, when the anatomical structure is clear in an image (eg, a template image)) according to aspects of the present disclosure;

[0022] Figure 6 A schematic diagram showing querying a region of interest using a mask (eg, when an anatomical structure is clear in an image (eg, a template image)) according to aspects of the present disclosure;

[0023] Figure 7 A schematic diagram showing a method for performing image region localization (eg, when an anatomical structure is unclear in an image (eg, a template image)) according to aspects of the present disclosure;

[0024] Figure 8 A schematic diagram showing a method for generating a template image (eg, using a clustering-based approach) according to various aspects of the present disclosure;

[0025] Fig. 9 A flow chart showing a method for performing one-time anatomical structure localization (eg, when the anatomical structure is not clear in an image (eg, a template image)) according to aspects of the present disclosure;

[0026] Fig.10 A schematic diagram showing segmentation of a pair of images according to various aspects of the present disclosure;

[0027] Fig.11 Associating anatomical labels to cluster label regions according to aspects of the present disclosure is shown;

[0028] Fig.12 illustrates positioning along orthogonal planes according to aspects of the present disclosure;

[0029] Fig.13 is an example of an image of a cluster of anatomically contiguous regions according to aspects of the present disclosure;

[0030] Fig.14 is a schematic diagram of a workflow process for concurrent longitudinal lesion detection and analysis by a user according to aspects of the present disclosure;

[0031] Fig.15 A flow chart showing a method for performing concurrent longitudinal lesion detection and analysis of a user according to aspects of the present disclosure is shown;

[0032] Fig.16 The present invention shows the use of various aspects of the present invention. Fig.15 The results of longitudinal lesion detection using the method in;

[0033] Fig.17 The present invention shows the use of various aspects of the present invention. Fig.15 The results of longitudinal lesion detection of brain lesions in axial T2 Flair data by the method in;

[0034] Fig.18 According to various aspects of the present disclosure Fig.14 a schematic diagram of a portion of a workflow process for a first point in time;

[0035] Fig.19 The present invention shows the use of various aspects of the present invention. Fig.14 The results of the second and third time points of the workflow process;

[0036] Fig. 20 The present invention shows the use of various aspects of the present invention. Fig.14 The results of the fourth and fifth time points of the workflow process;

[0037] Fig.21 The present invention shows the use of various aspects of the present invention. Fig.15 The results of longitudinal lesion detection of brain lesions in post-contrast T1 coronal data by the method in;

[0038] Fig. 22 The present invention shows the use of various aspects of the present invention. Fig.15 The results of longitudinal lesion detection of brain lesions in axial T2 Flair data by the method in;

[0039] Fig.23 The present invention shows the use of various aspects of the present invention. Fig.15 The results of longitudinal lesion detection of brain lesions in axial T2 Flair data by the method in; and

[0040] Fig.24 The present invention shows the use of various aspects of the present invention. Fig.15 The method in is used to perform the flipping and partial covering effects. DETAILED DESCRIPTION

[0041] One or more specific embodiments will be described below. In order to provide a concise description of these embodiments, not all features of an actual implementation are described in the specification. It should be understood that in the development of any such actual implementation, as in any engineering or design project, many implementation-specific decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints that may vary from implementation to implementation. In addition, it should be understood that such development efforts may be complex and time-consuming, but are still routine tasks for design, fabrication, and manufacturing for ordinary technicians who benefit from this disclosure.

[0042] When introducing elements of various embodiments of the present subject matter, the articles "a," "an," "the," and "said" are intended to indicate that there are one or more elements. The terms "comprising," "including," and "having" are intended to be inclusive, and mean that there may be additional elements in addition to the listed elements. Furthermore, any numerical examples in the following discussion are intended to be non-limiting, and thus the additional numerical values, ranges, and percentages are within the scope of the disclosed embodiments.

[0043] Although various aspects of the following discussion are provided in the context of medical imaging, it should be understood that the technology disclosed in the present invention is not limited to such medical contexts. In fact, the examples and explanations provided in such medical contexts are only for the convenience of explanation by providing examples of realistic specific implementations and applications. However, the technology disclosed in the present invention can also be used in other contexts, such as non-destructive inspection of manufactured parts or goods (i.e., quality control or quality review application scenarios) and / or image reconstruction of non-invasive inspection of packages, boxes, suitcases, etc. (i.e., security inspection or screening application scenarios). In general, the technology disclosed in the present invention can be used in any imaging or screening context or image processing or photography field, in which a group or class of collected data undergoes a reconstruction process to generate an image or volume.

[0044] The deep learning (DL) methods discussed herein may be based on artificial neural networks and may therefore encompass one or more of the following: deep neural networks, fully interconnected networks, convolutional neural networks (CNNs), unfolded neural networks, perceptrons, codecs, recurrent networks, wavelet filter banks, u-nets, generative adversarial networks (GANs), dense neural networks, or other neural network architectures. Neural networks may include shortcuts, activations, batch normalization layers, and / or other features. These techniques are referred to herein as DL techniques, although the term may also be used with particular reference to the use of deep neural networks, which are neural networks with multiple layers.

[0045] One type of deep learning model is a visual transformer model. A visual transformer model utilizes a transformer (e.g., a visual transformer) to perform image recognition tasks. Specifically, a visual transformer model decomposes an input image (e.g., a medical image) into blocks, processes the blocks using a transformer, and aggregates information for classification or object detection. A visual transformer model utilizes self-attention (i.e., a global operation) because it extracts information from the entire image. This enables the visual transformer model to effectively capture clear semantic correlations in an image. A visual transformer model obtains similar or better results than other types of deep learning models (e.g., convolutional networks) while requiring far fewer computational resources to train.

[0046] As discussed herein, DL techniques (which may also be referred to as deep machine learning, hierarchical learning, or deep structured learning) are a branch of machine learning techniques that employ mathematical representations of data and artificial neural networks for learning and processing such representations. For example, DL methods may be characterized as using one or more algorithms to extract or model highly abstract concepts of a class of data of interest. This may be accomplished using one or more processing layers, where each layer typically corresponds to a different level of abstraction, and thus may employ or utilize different aspects of the initial data or the output of the previous layer (i.e., a hierarchical or cascaded structure of layers) as the target of the process or algorithm of a given layer. In the context of image processing or reconstruction, this may be characterized as different layers corresponding to different feature levels or resolutions in the data. In general, the processing of one representation space to the next level representation space may be viewed as a "stage" of the process. Each stage of the process may be performed by a separate neural network or by different parts of a larger neural network.

[0047] The present disclosure provides systems and methods for performing one-time (e.g., single-pass) anatomical structure localization. Specifically, one-time anatomical structure localization is accomplished in conjunction with an unsupervised visual transformer for medical images (e.g., three-dimensional (3D) medical images). The technology disclosed in the present invention can be used for different types of medical images. For example, these images can be obtained from MRI, computed tomography (CT) imaging, or other types of imaging systems. In the present disclosure, the technology is described in the context of MRI.

[0048] The approach described herein combines unsupervised training (i.e., without ground truth) of a large amount of medical imaging data with one-time or single-pass labeling of the data using template pixels or regions. This enables automatic labeling of a large amount of data without any supervision. Labeling enables anatomical localization using labeling of specific regions or textures based on the template pixels used. In certain embodiments, labeling can be used for other applications (e.g., for lesion detection if a lesion can be found in the template image).

[0049] Using a visual transformer as the most important component, the visual transformer model is trained on an unlabeled data pool with the goal of obtaining a representation of the images, making them closer to similar images and separating the features of dissimilar images. The transformer architecture enables the acquisition of block-level features (e.g., interpolation) that can be expanded to pixel-level features. After completing the model training, the techniques described herein allow the selection of template images and regions of interest within the anatomical structure of interest. Different anatomical labeling methods can be used using template regions (e.g., template images) and pixel-level features obtained from unsupervised training. The same trained model can be used for all body parts.

[0050] In some embodiments, anatomical labeling is based on existing prominent regions of interest (e.g., in a template image). This approach involves directly querying features corresponding to regions of interest in a template image in a target image (e.g., a target medical image). This approach is preferred when landmarks are very clear, such as bones (e.g., bright, bulky, or prominent areas within dark edges).

[0051] In certain embodiments (e.g., when the anatomical structure is clear in the image (e.g., the template image)), the systems and methods disclosed herein (e.g., for performing one-time or single-pass anatomical structure localization) include obtaining a medical image of a subject at a processor. The systems and methods disclosed herein also include receiving a selection of a template image at the processor, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels, and a first reference point is marked on the template image. The systems and methods disclosed herein also include inputting both the medical image and the template image into a trained visual transformer model via the processor. The systems and methods disclosed herein even include outputting both the block-level feature portions and the image-level feature portions of both the medical image and the template image from the trained visual transformer model via the processor. The systems and methods disclosed herein also include interpolating the pixel-level feature portions from the block-level feature portions of both the medical image and the template image via the processor. The systems and methods disclosed herein further include clustering the pixel-level feature portions of both the medical image and the template image into anatomically similar regions via the processor. The systems and methods disclosed herein also include assigning cluster labels to pixels of both the medical image and the template image of corresponding anatomically similar regions via a processor. In certain embodiments, the systems and methods disclosed herein include assigning anatomical labels corresponding to the region of interest in the template image to corresponding pixel-level features in the medical image via a processor. In certain embodiments, the systems and methods disclosed herein include marking the region of interest in the template image with a first reference point via a processor. In certain embodiments, the systems and methods disclosed herein include marking the corresponding region of interest in the medical image with a second reference point via a processor, the second reference point corresponding to the region of interest in the template image with the first reference point.

[0052] In certain embodiments, the systems and methods disclosed herein include obtaining a set of orthogonal medical images of a subject at a processor, wherein the set of orthogonal medical images depicts a three-dimensional volume of a region of interest of the subject. The systems and methods disclosed herein also include receiving at the processor a selection of both a corresponding template image and a corresponding region of interest within the corresponding template image for use with each corresponding medical image in a set of orthogonal images, wherein each corresponding template image includes one or more anatomical landmarks assigned with corresponding anatomical labels. The systems and methods disclosed herein also include inputting both the set of orthogonal medical images and the corresponding template images into a trained visual transformer model via a processor. The systems and methods disclosed herein also include outputting both corresponding block-level features and corresponding image-level features of both a set of orthogonal medical images and the corresponding template images from the trained visual transformer model via a processor. The systems and methods disclosed herein further include interpolating corresponding pixel-level features of corresponding block-level features from both a set of orthogonal medical images and the corresponding template images via a processor. The system and method disclosed in the present invention also include locating and labeling corresponding pixel-level features in each corresponding medical image in a set of orthogonal images using corresponding pixel-level features in a corresponding region of interest in each corresponding template image via a processor.

[0053] In certain embodiments, anatomical labeling is based on clustering. For anatomical structures that are not clearly separated in an image (e.g., a template image), this approach utilizes pixel clustering based on pixel-level features. The number of clusters is visually fine-tuned as needed to separate image portions of desired granularity on the template image. In addition, the coordinates of the landmarks are marked on the template image. For any target image, paired clustering is performed on the pixel features of the target image and the template image to obtain corresponding regions in the two. The region corresponding to the landmark point (e.g., reference point) in the template image is transferred to the target image using the region correspondence. This effectively identifies the region of interest.

[0054] In certain embodiments (e.g., when the anatomical structure is not clear in the image), the systems and methods disclosed herein (e.g., for performing one-time anatomical structure localization) include obtaining a medical image of a subject at a processor. The systems and methods disclosed herein also include receiving a selection of a template image at the processor, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels, and a first reference point is marked on the template image. The systems and methods disclosed herein also include inputting both the medical image and the template image into a trained visual transformer model via a processor. The systems and methods disclosed herein even include outputting both the block-level feature parts and the image-level feature parts of both the medical image and the template image from the trained visual transformer model via a processor. The systems and methods disclosed herein also include interpolating the pixel-level feature parts from the block-level feature parts of both the medical image and the template image via a processor. The systems and methods disclosed herein also include clustering the pixel-level feature parts of both the medical image and the template image into anatomically similar regions via a processor. The systems and methods disclosed herein also include assigning cluster labels to pixels of both the medical image and the template image of corresponding anatomically similar regions via a processor. In certain embodiments, assigning cluster labels includes applying a segmentation mask to both the medical image and the template image. In certain embodiments, the systems and methods disclosed herein include assigning, via a processor, corresponding anatomical labels associated with corresponding anatomically similar regions in the template image to one or more corresponding anatomically similar regions in the medical image. In certain embodiments, the systems and methods disclosed herein include marking, via a processor, a region of interest in the template image with a first reference point. In certain embodiments, the systems and methods disclosed herein include marking, via a processor, corresponding regions in the medical image with a second reference point that corresponds to the region of interest in the template image marked with the first reference point.

[0055] In certain embodiments, the systems and methods disclosed herein include obtaining a set of orthogonal medical images of a subject at a processor, wherein the set of orthogonal medical images depicts a three-dimensional volume of a region of interest of the subject. The systems and methods disclosed herein include receiving a selection of a set of template images at a processor, wherein each template image in the set of template images includes one or more anatomical landmarks assigned with corresponding anatomical labels, and corresponding reference points are marked on each template image in the set of template images, wherein each template image in the set of template images corresponds to a corresponding medical image in the set of medical images. The systems and methods disclosed herein also include inputting both the set of orthogonal medical images and the set of template images into a trained visual transformer model via a processor. The systems and methods disclosed herein further include outputting both corresponding block-level features and corresponding image-level features of both the set of orthogonal medical images and the set of template images from the trained visual transformer model via a processor. The systems and methods disclosed herein further include interpolating corresponding pixel-level features from corresponding block-level features of both the set of orthogonal medical images and the set of template images via a processor. The system and method disclosed in the present invention even include clustering corresponding pixel-level features of both a set of orthogonal medical images and a set of template images into anatomically similar regions via a processor. The system and method disclosed in the present invention also include assigning clustering labels to pixels of both a set of orthogonal medical images and a set of template images of corresponding anatomically similar regions via a processor.

[0056] Clustering pixel-level features allows for customizable region segmentation granularity. The number of clusters and / or template pixel regions of interest need only be set once to perform one or more tasks at hand, and then automatically used to label large data pools (e.g., in target medical images).

[0057] The disclosed embodiments do not require manual annotation. The disclosed embodiments enable rapid automatic localization of anatomical landmarks (e.g., for subsequent imaging scans). The disclosed embodiments utilize a transformer to identify corresponding landmark points between pairs of images without supervision, image registration, or any standardized shape or size. The disclosed embodiments also save time planning scans by avoiding repeated localizer scans.

[0058] In addition, the present disclosure provides systems and methods for user concurrent longitudinal lesion detection and analysis using trained virtual transformer models. Specifically, the systems and methods disclosed in the present invention enable the use of an algorithm that is configured to automatically take a single time point or single slice marker of a lesion (e.g., with its own feature map) as input (e.g., user input on the subject's reference imaging data), and then identify similar lesions in each volume or across different time points and report relevant metrics of these lesions as part of the longitudinal analysis. Image registration of imaging data at different time points is not required. The embodiments disclosed in the present invention will segment lesions by automatically performing repetitive tasks across time points, thereby increasing the throughput of clinician readings, while also facilitating the identification of new dynamic lesions of similar characteristics. In addition, since relevant metrics are dynamically displayed to the user, disease velocity and progression can be easily determined.

[0059] In certain embodiments, the embodiments disclosed herein include obtaining medical imaging data of a subject, which is collected at a time point later than the collection of reference medical imaging data of the subject. The disclosed embodiments also include automatically marking non-lesion areas and one or more lesions in the medical imaging data based on one or more non-lesion areas and one or more lesions marked in the reference medical imaging data. The embodiments disclosed herein also include inputting the marked medical imaging data into a trained visual transformer model. The embodiments disclosed herein even include outputting a visual transformer feature from the trained visual transformer model based on one or more non-lesion areas and one or more lesion areas marked from the medical imaging data. The embodiments disclosed herein further include identifying similar feature voxels in the medical imaging data to identify one or more lesions based on the output visual transformer feature. The embodiments disclosed herein also include labeling one or more lesions in the medical imaging data using a segmentation mask.

[0060] Embodiments disclosed herein enable acceleration of longitudinal imaging workflows, particularly in oncology where imaging is used multiple times as part of a subsequent course of clinical diagnosis as well as treatment guidance and outcomes. Since the clinician marks one or more lesions of interest on reference imaging data (e.g., on one or more relevant slices of the imaging volume), the workflow reduces clinician fatigue from marking one or more lesions on each longitudinal data point. Additionally, automated reporting ensures rapid determination of disease progression or velocity to determine the impact of treatment. Additionally, since the user marks reference lesions, trust in the artificial intelligence solution is instilled by providing a collection of features that are marked or any missed in the analysis results of the longitudinal data.

[0061] Embodiments disclosed in the present invention also enable longitudinal data analysis without any supervised learning. Embodiments disclosed in the present invention also enable to adapt to any lesion type or heterogeneity by using the information of reference markers on single time point data, thereby ensuring that radiological knowledge and key points are incorporated into the process. Embodiments disclosed in the present invention even enable to dynamically update lesion information based on the change of lesion characteristics (e.g., habitat change, heterogeneity change, etc.). Embodiments disclosed in the present invention also enable to clean lesions (e.g., fine-tune masks) using a generalized segmentation model with prompts or artificial intelligence post-processing tools. Embodiments disclosed in the present invention also enable to track a given lesion type associated with a specific region or characteristic. Embodiments disclosed in the present invention also enable to report related tumor measurements, and push the data to a database (i.e., the status information of one or more lesions is always stored in the platform database) in text or graphical form. Embodiments disclosed in the present invention also include making the most relevant slices highlighted or displayed on a user display or user interface of a longitudinal analysis platform.

[0062] Considering the above, Figure 1 , a magnetic resonance imaging (MRI) system 100 is schematically illustrated as including a scanner 102, scanner control circuitry 104, and system control circuitry 106. According to embodiments described herein, the MRI system 100 is generally configured to perform MR imaging.

[0063] The system 100 also includes a remote access and storage system or device, such as a picture archiving and communication system (PACS) 108, or other devices, such as teleradiology equipment, that enable on-site or off-site access to data acquired by the system 100. In this way, MR data can be acquired and then processed and evaluated on-site or off-site. Although the MRI system 100 can include any suitable scanner or detector, in the illustrated embodiment, the system 100 includes a whole body scanner 102 having a housing 120 through which an aperture 122 is formed. An examination table 124 can be moved into the aperture 122 to allow a patient 126 (e.g., a subject) to be positioned therein to image selected anatomical structures within the patient's body.

[0064] The scanner 102 includes a series of associated coils for generating a controlled magnetic field for exciting gyromagnetic materials within the anatomical structure of the patient being imaged. Specifically, a primary magnetic coil 128 is provided for generating a primary magnetic field B0 that is generally aligned with the orifice 122. A series of gradient coils 130, 132, and 134 allow controlled gradient magnetic fields to be generated during an examination sequence for position encoding of certain gyromagnetic nuclei within the patient 126. A radio frequency (RF) coil 136 (e.g., an RF transmit coil) is configured to generate radio frequency pulses for exciting certain gyromagnetic nuclei within the patient. In addition to the coils that may be located locally at the scanner 102, the system 100 also includes a set of receiving coils or RF receiving coils 138 (e.g., a coil array) configured to be placed proximal to the patient 126 (e.g., against the patient). For example, the receiving coil 138 may include a cervical / thoracic / lumbar (CTL) coil, a head coil, a single-sided spine coil, etc. Generally, the receive coil 138 is placed near or on top of the patient 126 to receive the weak RF signals (weak relative to the transmit pulses generated by the scanner coil) generated by certain gyromagnetic nuclei in the patient's body when the patient 126 returns to his or her relaxed state.

[0065] The various coils of the system 100 are controlled by external circuitry to generate the required fields and pulses and to read the emission from the gyromagnetic material in a controlled manner. In the illustrated embodiment, the main power supply 140 provides power to the primary field coil 128 to generate the primary magnetic field Bo. The power input (e.g., power from a utility or grid), a power distribution unit (PDU), a power supply (PS), and a drive circuit 150 can provide power together to pulse the gradient field coils 130, 132, and 134. The drive circuit 150 may include an amplification and control circuit system for supplying current to the coils as defined by the digitized pulse sequence output by the scanner control circuit system 104.

[0066] Another control circuit 152 is provided for regulating the operation of the RF coil 136. The circuit 152 includes a switching device for alternating between an active mode of operation and a passive mode of operation, wherein the RF coil 136 transmits and does not transmit a signal, respectively. The circuit 152 also includes an amplification circuit system configured to generate RF pulses. Similarly, the receiving coil 138 is connected to a switch 154 that is capable of switching the receiving coil 138 between a receiving mode and a non-receiving mode. Thus, in the receiving mode, the receiving coil 138 resonates with the RF signal generated by the release of the gyromagnetic nuclei in the patient 126, and in the non-receiving mode, they do not resonate with the RF energy from the transmitting coil (i.e., the coil 136) to prevent undesired operation. In addition, the receiving circuit 156 is configured to receive data detected by the receiving coil 138, and may include one or more multiplexing and / or amplification circuits.

[0067] It should be noted that although the scanner 102 and control / amplification circuit system described above are illustrated as being coupled by a single line, in actual examples there may be many such lines. For example, separate lines may be used for control, data communication, power transmission, etc. In addition, appropriate hardware may be provided along each type of line for properly processing data and current / voltage. In practice, various filters, digitizers, and processors may be provided between the scanner and either or both of the scanner control circuit 104 and the system control circuit system 106.

[0068] As shown, the scanner control circuit system 104 includes an interface circuit 158 ​​that outputs signals for driving the gradient field coils and the RF coils and for receiving data representing magnetic resonance signals generated in the examination sequence. The interface circuit 158 ​​is coupled to a control and analysis circuit 160. Based on a defined scheme selected via the system control circuit 106, the control and analysis circuit 160 executes commands for driving the circuits 150 and 152.

[0069] The control and analysis circuitry 160 is also used to receive magnetic resonance signals and perform subsequent processing before transmitting the data to the system control circuitry 106. The scanner control circuitry 104 also includes one or more memory circuits 162 that store configuration parameters, pulse sequence descriptions, examination results, etc. during operation.

[0070] Interface circuit 164 is coupled to control and analysis circuit 160 for exchanging data between scanner control circuit system 104 and system control circuit system 106. In some embodiments, control and analysis circuit 160, although shown as a single unit, may include one or more hardware devices. System control circuit 106 includes interface circuit 166, which receives data from scanner control circuit system 104 and transmits data and commands back to scanner control circuit system 104. Control and analysis circuit 168 may include a CPU in a general or special computer or workstation. Control and analysis circuit 168 is coupled to memory circuit 170 to store programming code for operating MRI system 100, and to store processed image data for later reconstruction, display and transmission. The programming code may execute one or more algorithms that are configured to perform the reconstruction of acquisition data as described below when executed by a processor. In some embodiments, memory circuit 170 may store a visual transformer model for the technology described below. In some embodiments, image reconstruction may occur on a separate computing device having a processing circuit system and a memory circuit system.

[0071] Programming code (e.g., of an MRI system or another type of imaging system or a remote computing device) may enable one-time anatomical structure localization with an unsupervised visual transformer. In certain embodiments, the programming code may enable obtaining a medical image of a subject (e.g., when the anatomical structure is clear in an image (e.g., a template image)). The programming code may enable receiving a selection of a template image, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels, and a first reference point is marked on the template image. The programming code may enable both the medical image and the template image to be input into a trained visual transformer model. The programming code may enable outputting both block-level features and image-level features of both the medical image and the template image from the trained visual transformer model. The programming code may enable interpolation of pixel-level features from the block-level features of both the medical image and the template image. The programming code may enable clustering of pixel-level features of both the medical image and the template image into anatomically similar regions. The programming code may enable assigning clustering labels to pixels of both the medical image and the template image that correspond to anatomically similar regions. In some embodiments, the programming code may enable assigning anatomical labels corresponding to regions of interest in a template image to corresponding pixel-level features in the medical image. In some embodiments, the programming code may enable marking the regions of interest in the template image with a first reference point. In some embodiments, the programming code may enable marking the corresponding regions of interest in the medical image with a second reference point that corresponds to the regions of interest in the template image with the first reference point.

[0072] In certain embodiments, the programming code may enable obtaining a medical image of a subject (e.g., when anatomical structures are not clear in the image). The programming code may enable receiving a selection of a template image, wherein the template image includes one or more anatomical landmarks assigned with corresponding anatomical labels, and a first reference point is marked on the template image. The programming code may enable both the medical image and the template image to be input into a trained visual transformer model. The programming code may enable outputting both the block-level features and the image-level features of both the medical image and the template image from the trained visual transformer model. The programming code may enable interpolation of the pixel-level features of the block-level features from both the medical image and the template image. The programming code may enable clustering of the pixel-level features of both the medical image and the template image into anatomically similar regions. The programming code may enable assigning cluster labels to pixels of both the medical image and the template image that correspond to anatomically similar regions. In certain embodiments, assigning cluster labels includes applying a segmentation mask to both the medical image and the template image. In some embodiments, the programming code may enable assigning to one or more corresponding anatomically similar regions in the medical image corresponding anatomically similar regions associated with corresponding anatomically similar regions in the template image. In some embodiments, the programming code may enable marking of a region of interest in the template image with a first reference point. In some embodiments, the programming code may enable marking of a corresponding region in the medical image with a second reference point that corresponds to the region of interest in the template image marked with the first reference point.

[0073] In certain embodiments, the processing code may obtain medical imaging data of the subject, which is collected at a time point later than the reference medical imaging data of the subject. The processing code may also automatically mark the non-lesion area and one or more lesions in the subsequent medical imaging data based on the one or more non-lesion areas and one or more lesions marked in the reference medical imaging data. The processing code may also input the marked medical imaging data into the trained visual transformer model. The programming code may also output the visual transformer feature from the trained visual transformer model based on the marked one or more non-lesion areas and one or more lesion areas from the medical imaging data. The programming code may also identify similar feature voxels in the medical imaging data to identify one or more lesions based on the output visual transformer feature. The programming code may also label one or more lesions in the medical imaging data using a segmentation mask.

[0074] Additional interface circuitry 172 may be provided for exchanging image data, configuration parameters, etc. with external system components (such as remote access and storage device 108). Finally, system control and analysis circuitry 168 may be communicatively coupled to various peripheral devices for facilitating operator interface and generating hard copies of reconstructed images. In the illustrated embodiment, these peripheral devices include a printer 174, a monitor 176, and a user interface 178, which includes devices such as a keyboard, a mouse, a touch screen (e.g., integral to the monitor 176), and the like.

[0075] Figure 2 A schematic diagram of a visual transformer model 180 is shown. As shown, a medical image 182 is input into the visual transformer model. The medical image 182 is split into a series of non-overlapping blocks 184 of fixed size. The blocks 184 are flattened and then linearly embedded (i.e., arranged in a linear sequence and multiplied by an embedding matrix). At the same time, an absolute position embedding is added. A [CLS] token is also added as a representation of the entire image, which can be used for classification. The block and positioning embeddings produce a vector 186 (e.g., a learnable vector) that is fed to a transformer encoder 188. The transformer encoder 188 may include a plurality of blocks. Each block may include a normalization layer, a multi-head attention network, and a multi-layer perceptron. The normalization layer enables the visual transformer model 180 to adapt to changes in the blocks 184 (e.g., sequence tokens). The multi-head attention network generates an attention map (e.g., an attention head) based on a given embedded visual token. The multi-layer perceptron is a two-layer classification network with a Gaussian error linear unit (GELU) at the end. The final multi-layer perceptron block (e.g., MLP head 190) may be output from the transformer encoder 188. The MLP head 190 includes a feature portion 192 (e.g., an image feature portion and a block feature portion) from the attention head. The feature portion 192 is normalized as shown by reference numeral 194 to generate a vector as shown by reference numeral 196.

[0076] Figure 3 Schematic diagram of self-supervised training of the visual transformer model 180 is shown. Figure 3As shown, a large dataset 198 of unlabeled medical imaging data (e.g., multiple unlabeled medical images 200) is obtained. In some embodiments, the multiple medical images are MR images. In some embodiments, the multiple medical images can be obtained from other types of imaging (e.g., CT imaging). Each image 200 of the large dataset 198 is cropped and transformed to generate a cropped and transformed image version of each image, as shown by reference numeral 202. The cropped and transformed image 202 is input into the visual transformer model 180. Specifically, the cropped and transformed image 202 is input into both the student visual transformer 204 (e.g., student ViT) and the teacher visual transformer 206 (e.g., teacher ViT). Both the student visual transformer 204 and the teacher visual transformer 206 have the same architecture but different parameters. Both the student visual transformer 204 and the teacher visual transformer 206 output corresponding K-dimensional features or vectors (e.g., block features and image features) represented by reference numerals 208 and 210. The output of the teacher visual transformer 206 (e.g., K-dimensional features 210) is centered with the average value obtained by batch calculation. The outputs of both the student visual transformer 204 and the teacher visual transformer 206 (e.g., K-dimensional features 208 and 210) are each normalized with the temperature softmax for the feature dimension. The similarity between the K-dimensional features 208 and 210 is measured using cross entropy loss, as shown by arrow 212. The gradient is back-propagated only through the student visual transformer 204 (e.g., by applying a stop gradient operator on the teacher visual transformer 206), as shown by arrow 214. The parameters of the teacher visual transformer 206 are updated with the exponential moving average of the parameters of the student visual transformer 204, as shown by arrow 216. The teacher visual transformer 206 acts as a model ensemble outline for improving the performance of the model 180. Throughout the training period, the teacher visual transformer 206 has better performance than the student visual transformer 204 and thus guides the training of the student visual transformer 204 by providing target features of better quality.

[0077] The model 180 enables unsupervised learning of image features. The most important components of the model 180 are visual transformers (e.g., student visual transformer 204 and teacher visual transformer 206). The most important component of the visual transformer enables the model 180 to learn block-level features (e.g., based on token size). Post-processing (e.g., interpolation) enables the extraction of pixel-level features from the block-level features.

[0078] Figure 4 A flow chart of a method 218 for performing a one-time anatomical structure localization (eg, when the anatomical structure is clear in an image (eg, a template image)) is shown. One or more steps of the method 218 may be performed by Figure 1The processing circuitry of the magnetic resonance imaging system 100 in the embodiment of the present invention, the processing circuitry of another type of imaging system (e.g., a CT imaging system), or the processing circuitry of a separate computing device may be performed. One or more of the steps of method 218 may be performed simultaneously or in a sequential manner. Figure 4 The sequence shown is performed in a different order. Although discussed in the context of anatomical structure localization, the method 218 can also be used for lesion detection or other types of applications.

[0079] The method 218 includes obtaining a medical image of a subject (e.g., a target image) (block 220). The method also includes receiving a selection of both a template image and a region of interest (ROI) within the template image (block 222). In some embodiments, the selected region of interest is a mask. In some embodiments, the selected region of interest is a tile. Figure 8 The template image is generated. In certain embodiments, method 218 includes marking a region of interest in the template image with a first landmark or reference point (box 224). The template image includes one or more anatomical landmarks assigned with corresponding anatomical labels. Method 218 also includes inputting both the medical image and the template image into the trained visual transformer model (box 226). Method 218 even includes outputting both the block-level feature parts and the image-level feature parts of both the medical image and the template image from the trained visual transformer model (box 228). Method 218 also includes interpolating the pixel-level feature parts of the block-level feature parts from both the medical image and the template image (box 230). Method 218 also includes using the pixel-level feature parts within the region of interest of the template image to locate and label the corresponding pixel-level feature parts in the medical image (box 232). The corresponding pixel-level feature parts in the medical image can be labeled via a segmentation mask.

[0080] The method 218 also includes assigning an anatomical label corresponding to the region of interest in the template image to the corresponding pixel-level feature in the medical image (block 234). The method 218 even includes marking, via the processor, the corresponding region of interest in the medical image with a second reference point that corresponds to the region of interest in the template image with the first reference point (block 236).

[0081] Figure 5A schematic diagram of querying a region of interest using tiles (e.g., when the anatomical structure is clear in an image (e.g., a template image)) is shown. A template image 238 (of a shoulder region) is selected. A region of interest 240 (in the form of tiles) is selected in the template image 238. The selected region of interest 240 is a bone. The region of interest 240 (e.g., tiles) is used to query a target medical image 242 (of a shoulder region) using pixel-level features obtained by inputting both the template image 238 and the target medical image 242 into a trained visual transformer model. Pixel-level features in the target medical image 242 that correspond to pixel-level features in the region of interest 240 of the template image 238 are located and labeled (e.g., via a segmentation mask), as shown by reference numeral 244. The template image 238 and the target medical image 242 are MR images.

[0082] Figure 6 A schematic diagram of querying a region of interest using a mask (e.g., when the anatomical structure is clear in an image (e.g., a template image)) is shown. A template image 245 (of a shoulder region) is selected. A region of interest 246 (in the form of a mask) is selected in the template image 245. The selected region of interest 246 is a bone. Using pixel-level features obtained by inputting both the template image 245 and the target medical images 248, 250 into a trained visual transformer model, the region of interest 246 is used to query the target medical images 248, 250 (of the shoulder region) (e.g., in blocks). Pixel-level features in the target medical images 248, 250 that correspond to pixel-level features in the region of interest 246 of the template image 245 are located and labeled (e.g., via a segmentation mask), as shown by reference numerals 252, 254, respectively. The template image 245 and the target medical images 248, 250 are MR images.

[0083] Figure 7A schematic diagram for performing image region localization (e.g., when the anatomical structure is not clear in the image (e.g., template image)) is shown. The top row 255 of images 256, 258, 260 represents the coronal image, sagittal image, and axial image of the prostate region selected for the template image. The bottom row 262 of images 264, 266, 268 represents the coronal image, sagittal image, and axial image of the prostate region used as the target image. Both the template image and the target image are MR images. The selected region of interest in the template image (shown by the segmentation mask) is represented by the reference numeral 270. The corresponding region in the target image (shown by the segmentation mask) is represented by the reference numeral. With respect to anatomical structures (such as the prostate) at close range, localization may deviate from querying using the region of interest. For example, the localization in both the template image and the target image is always below the prostate. In these cases, the localization can be adjusted with standardized correction to obtain the prostate region of interest.

[0084] Figure 8 A schematic diagram for generating a template image (eg, using a clustering-based approach) is shown. Figure 8 As shown, medical image 274 is divided into blocks 276. Medical image 274 is an MR image of the prostate. Blocks 276 are input into the trained visual transformer model 180. Figure 3 The self-supervised learning trains the visual transformer model 180. The trained visual transformer model 180 outputs both the image-level feature portion 278 and the block-level feature portion 280 of the medical image 274. The pixel-level feature portion 282 (e.g., the corresponding vector obtained from each pixel) is obtained from the block-level feature portion 280 of the medical image 274 via interpolation, as shown by arrow 284. As shown by arrow 286, the pixel-level feature portion 282 is clustered (e.g., pixel clustering) so that the pixel-level feature portions 282 belonging to similar regions are clustered together, as shown in Figure 288. Then, different cluster labels are assigned to pixels in different regions as segmentation masks, as shown in image 290. The segmentation masks for different regions are color-coded with different colors. At this time, the cluster labels are not associated with the corresponding anatomical labels. The image 290 is judged (e.g., visually analyzed) as to whether the desired anatomical structures are well separated, as shown in figure 292. If the desired anatomical structure is not well separated, an appropriate number of clusters is selected or determined (as shown in reference numeral 294) and pixel clustering is repeated. If the desired anatomical structure is well separated, corresponding anatomical labels (as shown in reference numeral 296) may be assigned to the desired anatomical structure or one or more regions of interest to generate a template image.

[0085] Fig. 9A flow chart of a method 298 for performing one-time anatomical structure localization (eg, when the anatomical structure is not clear in an image (eg, a template image)) is shown. One or more steps of the method 298 may be performed by Figure 1 The method 298 may be performed by the processing circuitry of the magnetic resonance imaging system 100 in the embodiment of the present invention, the processing circuitry of another type of imaging system (e.g., a CT imaging system), or the processing circuitry of a separate computing device. One or more of the steps of the method 298 may be performed simultaneously or in a sequential manner. Fig. 9 The sequence shown is performed in a different order. Although discussed in the context of anatomical structure localization, method 298 can also be used for lesion detection or other types of applications.

[0086] Method 298 includes obtaining a medical image (eg, a target image) of a subject (block 300). Method 298 also includes receiving a selection of a template image (block 302). Figure 8 The template image is generated. In some embodiments, method 298 includes marking a region of interest (ROI) or a landmark area in the template image with a first landmark or reference point (box 304). The template image includes one or more anatomical landmarks assigned with corresponding anatomical labels. In some embodiments, the region of interest or the landmark area can be marked after pairwise clustering of the pixel-level features in the medical image and the template image. Method 298 also includes inputting both the medical image and the template image into the trained visual transformer model (box 306). Method 298 even includes outputting both the block-level feature parts and the image-level feature parts of the medical image and the template image from the trained visual transformer model (box 308). Method 298 also includes interpolating the pixel-level feature parts of the block-level feature parts from both the medical image and the template image (box 310). Method 298 also includes performing pairwise clustering to cluster the pixel-level feature parts of both the medical image and the template image into anatomically similar regions (box 312). The method 298 even further includes assigning cluster labels to pixels of both the medical image and the template image of corresponding anatomically similar regions (box 314). Corresponding pixels in both the medical image and the template image may be labeled via the segmentation mask. The method 298 also includes assigning corresponding anatomical labels associated with corresponding anatomically similar regions in the template image to one or more corresponding anatomically similar regions in the medical image (box 316). The method 298 also includes marking the corresponding regions in the medical image with a second reference point that corresponds to the region of interest in the template image marked with the first reference point (box 318).

[0087] Fig.10 A schematic diagram of segmenting a pair of images is shown. Figure 8As shown, a first image 320 (e.g., a template image) and a second image 322 (e.g., a medical image or a target image) are input into the trained visual transformer model 180. The first image 320 and the second image 322 are MR images of the prostate. Although not shown, both the first image 320 and the second image 322 are each separated into blocks that are input into the trained visual transformer model 180. Figure 3 The self-supervised learning trains the visual transformer model 180. The trained visual transformer model 180 outputs both image-level features and block-level features of both the first image 320 and the second image 322. Corresponding pixel-level features 324, 326 (e.g., corresponding vectors obtained from each pixel) are obtained from the block-level features of both the first image 320 and the second image 322 via interpolation, as shown by arrow 284. As shown by arrow 328, the pixel-level features 324, 326 are clustered together (e.g., clustered pixels) so that the pixel-level features 282 belonging to similar regions are clustered together, as shown in Figure 330. Different cluster labels are then assigned to pixels in different regions as segmentation masks, as shown in corresponding images 332, 334 for the first image 320 and the second image 322. Pairs of corresponding regions appear in both images 322, 334. The segmentation masks for different regions are color-coded with different colors. The fat region, background, and pelvic region are similar in both images 322 and 324. Semantically similar regions are assigned similar cluster labels. Fig.10 As shown, even if the fields of view, scale, or intensity changes are different, feature parts belonging to similar regions are clustered on different images.

[0088] Fig.11 Associating anatomical labels to cluster label regions is shown. Fig.11 Shown from Fig.10 A first image 322 (eg, a template image) and a corresponding cluster labeled image 332. As shown in FIG. Fig.11 Also shown from Fig.10 In some embodiments, a second image 324 (e.g., a target image) and a corresponding cluster labeled image 334 are provided. In some embodiments, regions of interest or landmark regions in the template image 322 are labeled with reference points 336. Regions in the cluster labeled image 334 of the target image 324 are anatomically labeled with anatomical labels associated with the reference points 336 in the template image 322. In some embodiments, regions in the target image 324 corresponding to the reference points 336 in the template image 322 are labeled with reference points corresponding to the reference points 336.

[0089] Fig.12Positioning along orthogonal planes (eg, coronal, sagittal, and axial planes) is shown. Fig.12 The images shown in are MR images of the prostate. The top row 338 includes image 340, which is a template image labeled with reference point 342. Image 344 is a cluster labeled image of image 340. Image 346 is a target image. Image 348 is a cluster labeled image of image 346. Image 350 is a target image with reference point 352 transferred thereto, which corresponds to reference point 342 in the region of interest in the template image (i.e., image 340). Reference point 352 is located at a position equivalent to the centroid of the label mask for the corresponding region of interest in image 348. Images 340, 344, 346, 348, and 350 are along the axial plane. Images 340, 344, 346, and 348 are Fig.11 Corresponding to images 320, 322, 332 and 334 in.

[0090] Middle row 354 includes image 356, which is a template image labeled with reference point 358. Image 360 ​​is a cluster labeled image of image 356. Image 362 is a target image. Image 364 is a cluster labeled image of image 362. Image 365 is a target image with reference point 366 transferred thereto, which corresponds to reference point 358 in the region of interest in the template image (i.e., image 356). Reference point 366 is located at a position equivalent to the centroid of the label mask for the corresponding region of interest in image 364. Images 356, 360, 362, and 364 and 365 are along the sagittal plane.

[0091] The bottom row 368 includes image 370, which is a template image labeled with reference point 372. Image 374 is a cluster labeled image of image 370. Image 376 is a target image. Image 378 is a cluster labeled image of image 376. Image 380 is a target image with reference point 382 transferred thereto, which corresponds to reference point 372 in the region of interest in the template image (i.e., image 370). Reference point 382 is located at a position equivalent to the centroid of the label mask for the corresponding region of interest in image 378. Images 370, 374, 376, 378, and 380 are along the coronal plane. Fig.12 It is shown that 3D localization (which describes a 3D volume) is performed by generating local landmarks in a set of orthogonal images. These planes can be straight or inclined.

[0092] Fig.13384, 386, 388 are examples of clusters of anatomically contiguous regions. Image 384 is a breast region. Image 386 is a brain region. Image 388 is a spine / fat region. Clusters of anatomically contiguous regions are automatically generated using a transformer vision model trained using unsupervised learning, followed by clustering and cluster labeling. As shown in images 384, 386, 388, these regions have been self-localized. Image 384 of the breast region has been automatically anatomically labeled using a template image with breast labels, as shown in reference numeral 390.

[0093] Fig.14 196 is a schematic diagram of a workflow process for performing concurrent longitudinal lesion detection and analysis for a user. In the first module 400 for feature generation, the unsupervised visual transformer model 180 is pre-trained as described above. Specifically, the visual transformer model 180 is trained using self-supervised learning on multiple unlabeled medical images or imaging data 200 (i.e., a large amount of multi-modal, multi-contrast imaging data across various subjects). The virtual transformer feature parts (e.g., image feature parts and block feature parts) output by the virtual transformer model 180 are normalized to generate a vector 196.

[0094] In a second module 402 for processing a reference time point (e.g., Tp#1), reference medical imaging data 404 (e.g., a reference imaging volume) of a subject is obtained. Input is provided regarding one or more relevant slices 406 of the reference medical imaging data 404 via user interaction (e.g., the user uses an input device to mark). The user input includes providing a mark or hint (e.g., represented by an asterisk 408) of one or more lesions on the one or more relevant slices 406 of the reference medical imaging data 404. The user input includes providing a mark or hint (e.g., represented by an X 410) of one or more non-lesion areas on the one or more relevant slices 406 of the reference medical imaging data 404. The marks or hints 408, 410 mark the features to be utilized with the trained visual transformer model 180. The marked reference medical imaging data 404 is input into the trained visual transformer model 180. The trained visual transformer model 180 outputs a visual transformer feature 412 from the trained visual transformer model 180 based on the labeled one or more non-lesion regions and one or more lesions from the reference medical imaging data 404. The output visual transformer feature 412 is used to identify similar feature voxels in the reference medical imaging data 404 to identify one or more lesions in the reference medical imaging data 404. For example, feature correction or clustering based on user input or compact machine learning input may be used to identify similar feature voxels in the reference medical imaging data 404. The segmentation mask (e.g., as mask 414) is used to label one or more lesions within the reference medical imaging data 404. The mask 414 on the reference medical imaging data 404 may be further refined (e.g., via a generalized segmentation refinement model).

[0095] In a third module 416 for processing data across subsequent time points (e.g., Tp#n) or longitudinal data, subsequent medical imaging data 418 (e.g., subsequent imaging volumes) of the subject are obtained at one or more time points later than the reference time point. n-1), automatically marking the one or more non-lesion areas and one or more lesions in the subsequent medical imaging data 418. When automatically marking, a mark or prompt (e.g., represented by an asterisk 420) of the one or more lesions is placed on the subsequent medical imaging data 418. In addition, a mark or prompt (e.g., represented by an X 422) of the one or more non-lesion areas is placed on the subsequent medical imaging data 418. The marks or prompts 420, 422 mark the features to be used with the trained visual transformer model 180. In some embodiments, a user may provide input to adjust the marking of the one or more non-lesion areas and / or the one or more lesions. In some embodiments, the reference imaging data may be histogram matched with the subsequent medical imaging data before the marked subsequent medical imaging data is input into the trained visual model 180. The marked subsequent medical imaging data 418 is input into the trained visual transformer model 180. The trained visual transformer model 180 outputs a visual transformer feature 424 from the trained visual transformer model 180 based on the labeled one or more non-lesion regions and one or more lesions from the subsequent medical imaging data 418. The output visual transformer feature 424 is used to identify similar feature voxels in the subsequent medical imaging data 418 to identify one or more lesions in the subsequent medical imaging data 418. For example, feature correction or clustering based on user input or compact machine learning input may be used to identify similar feature voxels in the subsequent medical imaging data 418. One or more lesions within the subsequent medical imaging data 418 are labeled using a segmentation mask (e.g., as a mask 426). The mask 426 on the subsequent medical imaging data 418 may be further refined (e.g., via a generalized segmentation refinement model).

[0096] One or more lesions labeled with masks in both reference medical imaging data 404 and subsequent medical imaging data 418 are measured. For example, arrow 428 indicates the measurement result of the diameter of the lesion in subsequent medical imaging data 418. The measurement may be related to the diameter of the lesion, the volume of the lesion, the density of the lesion or other measurements of the lesion. In certain embodiments, the measurement may be related to the radioactive feature or other radioactive feature associated with the lesion habitat, the heterogeneity of the lesion, the texture of the lesion. A report on the change of the corresponding measurement of one or more lesions between the reference medical imaging data and the subsequent imaging data over time can be generated. These changes can be presented in graphical form, as shown in graph 430 (e.g., showing the change of lesion diameter over time) and graph 432 (e.g., showing the change of texture over time). Alternatively or in addition thereto, these changes can be characterized in a textual manner, as shown in figure numeral 434.

[0097] The status information of one or more lesions is always saved in the database. In addition, the workflow does not assume that the imaging data is geometrically synchronized or registered across data time points to determine the lesion characteristics across time points. In addition, since the user (e.g., clinician) marks the lesions at the reference time points, the algorithm can be easily extended to any type of lesion, making it a general tool for deployment on longitudinal tools or platforms.

[0098] Fig.15 A flow chart of a method 436 for performing concurrent longitudinal lesion detection and analysis of a user is shown. One or more steps of the method 436 may be performed by Figure 1 The method 436 may be performed by the processing circuitry of the magnetic resonance imaging system 100 in the embodiment of the present invention, the processing circuitry of another type of imaging system (e.g., a CT imaging system), or the processing circuitry of a separate computing device. One or more of the steps of the method 436 may be performed simultaneously or in a sequential manner. Fig.15 The order shown is different from the order in which it is executed.

[0099] The method 436 includes obtaining reference medical imaging data (e.g., a reference medical imaging volume) acquired from the subject (block 438). The reference medical imaging data can be used as an initial time point for a longitudinal analysis study. The method 436 also includes receiving user input that labels one or more non-lesion regions and one or more lesions of the reference medical imaging data (block 440). The method 436 also includes the labeled reference medical imaging data into the trained visual transformer model (e.g., Fig.14The method 436 further includes outputting visual transformer features (e.g., image-level features and / or pixel-level features) from the trained visual transformer model based on one or more labeled non-lesion regions and one or more lesions from the reference medical imaging data (box 444). The method 436 also includes identifying similar feature voxels (e.g., angiographic similarity) in the reference medical imaging data based on the output visual transformer features based on the reference medical imaging data to identify one or more lesions (box 446). For example, similar feature voxels in the reference medical imaging data may be identified using feature correction or clustering based on user input or compact machine learning input. The method 436 even includes labeling one or more lesions within the reference medical imaging data using a segmentation mask (box 448). In certain embodiments, the method 436 also includes automatically refining the segmentation mask using a post-processing scheme such as a generalized segmentation refinement model (box 450). The identified similar voxel features are used as prompts for the generalized segmentation refinement model. For example, a level set, CascadePSP, or Segment Anything Model may be utilized. Method 436 includes measuring one or more corresponding measures of one or more lesions in the reference medical imaging data (block 452). The measure may relate to the diameter of the lesion, the volume of the lesion, the density of the lesion, or other measures of the lesion. In certain embodiments, the measure may relate to radioactive features associated with the habitat of the lesion, the heterogeneity of the lesion, the texture of the lesion, or other radioactive features.

[0100] Method 436 includes obtaining subsequent medical imaging data of the subject, which is collected at a time point later than the time point when the reference medical imaging data of the subject is collected (box 454). Method 436 also includes automatically marking one or more non-lesion areas and one or more lesions in the subsequent medical imaging data based on the features marked in the reference medical imaging data (box 456). In some embodiments, method 436 includes receiving additional user input to update the marking of one or more non-lesion areas and one or more lesions in the subsequent medical imaging data after automatically marking the one or more non-lesion areas and one or more lesions in the subsequent medical imaging data (box 458). In some embodiments, method 436 includes performing preprocessing such as histogram matching between the reference imaging data and the subsequent medical imaging data before inputting the marked subsequent medical imaging data into the trained visual transformer model (box 460). Method 436 also includes inputting the marked subsequent medical imaging data into the trained visual transformer model (box 462). The method 436 even includes outputting visual transformer features (e.g., image-level features and / or pixel-level features) from the trained visual transformer model based on the labeled one or more non-lesion regions and one or more lesions from the subsequent medical imaging data (box 464). The method 436 also includes identifying similar feature voxels (e.g., angiographic similarity) in the subsequent medical imaging data based on the output visual transformer features to identify one or more lesions (box 466). For example, similar feature voxels in the reference medical imaging data can be identified using feature correction or clustering based on user input or compact machine learning input. The method 436 also includes labeling one or more lesions in the subsequent medical imaging data using a segmentation mask (box 468). In some embodiments, the method 436 also includes automatically refining the segmentation mask using a post-processing scheme such as a generalized segmentation refinement model (box 470). For example, a level set, CascadePSP, or a segmentation everything model can be used. The identified similar voxel features are used as a hint for the generalized segmentation refinement model. The method 436 also includes measuring one or more corresponding measurements of one or more lesions in the reference medical imaging data (block 472). The measurement may relate to the diameter of the lesion, the volume of the lesion, the density of the lesion, or other measurements of the lesion. In certain embodiments, the measurement may relate to radioactive features related to the habitat of the lesion, the heterogeneity of the lesion, the texture of the lesion, or other radioactive features.

[0101] Method 436 includes generating a report of changes over time of corresponding measurements of one or more lesions between the reference medical imaging data and the subsequent imaging data (box 474). The report may characterize the changes in one or more graphs. Alternatively or in addition, the report may be characterized in a textual manner. Method 436 includes displaying the report to a user on a display or graphical user interface (box 476). Method 436 also includes displaying the most relevant slices or images labeled with lesions for each time point (e.g., the reference medical imaging data and the subsequent medical imaging data acquired at different time points) (box 478). Boxes 454 to 478 may be repeated for subsequent medical imaging data acquired at different subsequent time points.

[0102] Fig.16 Shows the use of Fig.15 The results of longitudinal lesion detection by method 436 in the present invention are shown in FIG. Images (or slices) 480, 482, 484, 486, and 488 are derived from medical imaging data of a subject acquired at different sequential time points. Images 480, 482, 484, 486, and 488 represent time point 1, time point 2, time point 3, time point 4, and time point 5, respectively. Image 480 is used as a reference. Fig.16 As shown, image 480 is labeled by a user, wherein asterisks 490 mark lesions and X 492 mark non-lesion areas. Corresponding masks 494 represent lesions tracked in images 480, 482, 484, 486, and 488 at different time points.

[0103] Fig.17 Shows the use of Fig.15 The method 436 in the embodiment of the present invention performs longitudinal lesion detection on brain lesions in axial T2 Flair data. Images 496, 498 and 500 are derived from medical imaging data (MRI data) of the brain of a subject acquired at different sequential time points. Images 496, 498 and 500 represent time point 1, time point 2 and time point 3, respectively. Image 496 is used as a reference. Fig.17 As shown, image 496 is marked by a user, where asterisk 502 marks a lesion. Corresponding mask 504 represents the lesions in image 506 and image 508 that are the same as image 496 and image 498, respectively. Image 510 is the same as image 500. As shown, the brain is free of the lesions in image 500 and image 510. Images 512, 514, and 516 are feature map images corresponding to the imaging data in images 496, 498, and 500, respectively.

[0104] Fig.18 yes Fig.14Schematic diagram of a portion of a workflow process for a first time point (e.g., time point 1) in . Reference medical imaging data 518 (e.g., a reference imaging volume) of a subject is obtained. Input about one or more relevant slices of the reference medical imaging data 518 is provided via user interaction (e.g., the user uses an input device to mark). The user input includes providing a mark or prompt (e.g., represented by point 520) of a lesion on one or more relevant slices of the reference medical imaging data 518. The user input includes providing a mark or prompt (e.g., represented by point 522) of a non-lesion area on one or more relevant slices of the reference medical imaging data 518. The marks or prompts 520, 522 mark the feature parts (non-lesion areas and lesion areas) to be used together with the trained visual transformer model 180 (e.g., computer vision model DINOv2). The marked reference medical imaging data 518 is input into the trained visual transformer model 180. The trained visual transformer model 180 outputs visual transformer features 524 (e.g., image-level features 526 and block-level features 528) from the trained visual transformer model 180 based on the labeled one or more non-lesion regions and one or more lesions from the reference medical imaging data 518. The pixel-level features 530 from the block-level features 528 may be interpolated. The pixel-level features 530 are used to identify pixels that are similar to the lesion features (e.g., positive pixel features) marked by point 520 in the reference medical imaging data 518 and are dissimilar to the negative pixel features (e.g., marked by point 522) to identify one or more lesions in the reference medical imaging data 518. For example, feature correction or clustering based on user input or compact machine learning input may be used to identify similar feature voxels in the reference medical imaging data 518. One or more lesions within the reference medical imaging data 518 are labeled using a segmentation mask (e.g., as mask 532). As shown in reference numeral 534, it may be determined whether the segmentation is good enough for a rough segmentation. If not, additional positives (lesions) and / or negatives (non-lesion points) as indicated on the reference medical imaging data 518 may be selected, or the threshold may be adjusted, as shown in reference numeral 536. The mask 532 on the reference medical imaging data 518 may be further refined via a generalized segmentation refinement model 538 (e.g., CascadePSP or Segment Arbitrary Model (SAM)). The identified similar voxel features are used as hints for the generalized segmentation refinement model.

[0105] Fig.19 and Fig. 20 Shows the use of Fig.14 The workflow processes the results from imaging data acquired from the same subject at subsequent time points. Fig.19Images 540 , 542 , and 544 in represent the original medical imaging data, the pixel similarity segmentation applied to the original medical imaging data, and the refinement of the pixel similarity segmentation for time point 2 . Fig.19 Images 546 , 548 , and 550 in represent the original medical imaging data, the pixel similarity segmentation applied to the original medical imaging data, and the refinement of the pixel similarity segmentation for time point 3 . Fig. 20 Images 552 , 554 , and 556 in represent the original medical imaging data, the pixel similarity segmentation applied to the original medical imaging data, and the refinement of the pixel similarity segmentation for time point 4 . Fig. 20 Images 558, 560, and 562 in represent the original medical imaging data, the pixel similarity segmentation applied to the original medical imaging data, and the refinement of the pixel similarity segmentation for time point 5. Mask 564 indicates the lesions tracked at different time points.

[0106] Fig.21 Shows the use of Fig.15 The method in performs longitudinal lesion detection on brain lesions in T1 coronal data after contrast. Image 566 and image 568 are different slices of original medical imaging data (e.g., MRI data) of the brain of the subject collected at a first time point (e.g., time point 1). Image 570 and image 572 correspond to image 566 and image 568 and have a segmentation mask 574 for identifying brain lesions. Image 576 and image 578 are different slices of original medical imaging data (e.g., MRI data) of the brain of the subject collected at a second time point (e.g., time point 2). Image 580 and image 582 correspond to image 576 and image 578 and have a segmentation mask 574 for identifying brain lesions.

[0107] Fig. 22 Shows the use of Fig.15 The method 436 in axial T2 Flair data performs longitudinal lesion detection on brain lesions in longitudinal lesion detection. Image 584 is raw medical imaging data (e.g., MRI data) of the brain of the subject collected at a first time point (e.g., time point 1). Image 586 corresponds to image 584 and has a segmentation mask 587 that identifies brain lesions. Image 588 is raw medical imaging data (e.g., MRI data) of the brain of the subject collected at a second time point (e.g., time point 2). Image 590 corresponds to image 588 and has a segmentation mask 587 that identifies brain lesions.

[0108] Fig.23 Shows the use of Fig.15The result of longitudinal lesion detection of brain lesions in axial T2 Flair data by method 436 in . Image 592 and image 594 are different slices of original medical imaging data (e.g., MRI data) of the brain of the subject acquired at a first time point (e.g., time point 1). Segmentation mask 596 identifies brain lesions in image 592 and image 594. Image 598 and image 600 are different slices of original medical imaging data (e.g., MRI data) of the brain of the subject acquired at a second time point (e.g., time point 2). Segmentation mask 596 identifies brain lesions in image 598 and image 600. Image 602 and image 604 are different slices of original medical imaging data (e.g., MRI data) of the brain of the subject acquired at a third time point (e.g., time point 3). Image 606 and image 608 are different slices of original medical imaging data (e.g., MRI data) of the brain of the subject acquired at a fourth time point (e.g., time point 4). There are no brain lesions at time points 3 and 4.

[0109] Fig.24 Shows the use of Fig.15 The method 436 in the present invention performs the flipping and partial covering effect. Images 565, 566 and 568 are original medical imaging data (e.g., axial T2 Flair data of the brain of the subject acquired). Image 565 is the original medical imaging data used as a reference or initial time point. Image 566 is a flipped reference image. Image 568 is a translated and cropped reference image. Mask 570 marks the detected lesions. Flipping and partial covering are not correct. Fig.15 The methods in this paper have an impact.

[0110] Technical effects of the subject matter disclosed herein include eliminating the need for manual annotation. Technical effects of the subject matter disclosed herein also include enabling rapid automatic localization of anatomical landmarks (e.g., for subsequent imaging scans). Technical effects of the subject matter disclosed herein also include utilizing a transformer to identify corresponding landmark points between pairs of images without supervision, image registration, or any standardized shape or size. Technical effects of the subject matter disclosed herein even include saving time in planning scans by avoiding repeated localizer scans.

[0111] The technical effects of the subject matter disclosed in the present invention also include providing an unsupervised deep learning-based methodology for automatic longitudinal lesion segmentation and tracking by utilizing user input on a reference scan. The technical effects of the subject matter disclosed in the present invention also include enabling acceleration of the longitudinal imaging workflow, particularly in oncology where imaging is actually used multiple times as part of a subsequent course of treatment for clinical diagnosis and treatment guidance and outcomes. Because the clinician marks one or more lesions of interest on the reference imaging data (e.g., on one or more relevant slices of the imaging volume), the workflow reduces the clinician's fatigue caused by marking one or more lesions on each longitudinal data point. The technical effects of the subject matter disclosed in the present invention also include enabling automatic reporting to ensure rapid determination of disease progression or velocity to determine the impact of treatment.

[0112] Reference is made to the technology presented and claimed herein and applied to physical and specific examples of a practical nature that clearly improves upon the present state of the art and, therefore, is not abstract, intangible, or purely theoretical. In addition, if any claim appended to the end of this specification contains one or more elements designated as "means for [performing] the function of ..." or "steps for [performing] the function of ...," it is intended that such elements be interpreted under 35 U.S.C. § 112(f). However, for any claim containing elements designated in any other manner, it is not intended that such elements be interpreted under 35 U.S.C. § 112(f).

[0113] This written description uses examples to disclose the subject matter, including the best mode, and also to enable those skilled in the art to practice the subject matter, including making and using any devices or systems and performing any included methods. The patentable scope of the subject matter is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insignificant differences from the literal language of the claims.

Claims

1. A computer-implemented method for performing concurrent longitudinal lesion detection and analysis, the computer-implemented method comprising: obtaining medical imaging data of a subject, the medical imaging data being acquired at a time point later than that at which reference medical imaging data of the subject is acquired; automatically marking one or more non-lesion regions and one or more lesions in the medical imaging data based on the one or more non-lesion regions and one or more lesions marked in the reference medical imaging data; Inputting the labeled medical imaging data into the trained visual transformer model; as well as outputting a visual transformer feature from the trained visual transformer model based on the labeled one or more non-lesion regions and a plurality of lesions from the medical imaging data; Based on the outputted visual transformer features, identifying similar feature voxels in the medical imaging data to identify one or more lesions; and The one or more lesions within the medical imaging data are labeled using a segmentation mask.

2. The computer-implemented method of claim 1 , further comprising: obtaining the reference medical imaging data; receiving a user input, the user input marking the one or more non-lesion regions and the one or more lesions of the reference medical imaging data; inputting the labeled reference medical imaging data into the trained visual transformer model; outputting a visual transformer feature from the trained visual transformer model based on the one or more labeled non-lesion regions and the one or more lesions from the reference medical imaging data; Based on the outputted visual transformer features based on the reference medical imaging data, identifying similar feature voxels in the reference medical imaging data to identify the one or more lesions; as well as The one or more lesions within the reference medical imaging data are labeled using the segmentation mask.

3. The computer-implemented method of claim 2, further comprising measuring one or more corresponding measurements of the one or more lesions in both the reference medical imaging data and the medical imaging data.

4. The computer-implemented method of claim 3, further comprising generating a report regarding changes over time in the corresponding measurements of the one or more lesions between the reference medical imaging data and the imaging data.

5. The computer-implemented method of claim 2, further comprising performing histogram matching on the reference imaging data and the medical imaging data before inputting the medical imaging data into the visual transformer model.

6. The computer-implemented method of claim 1, wherein identifying the similar voxel features comprises utilizing feature correction or clustering or utilizing a compact machine learning model based on the user input.

7. The computer-implemented method of claim 1, further comprising automatically refining the segmentation mask.

8. The computer-implemented method of claim 7, wherein automatically refining the segmentation mask comprises utilizing a generalized segmentation refinement model that utilizes the identified similar voxel features.

9. The computer-implemented method of claim 2, further comprising, after automatically marking the one or more non-lesion areas and the one or more lesions in the medical imaging data, receiving additional user input to update the markings of the one or more non-lesion areas and the one or more lesions in the medical imaging data.

10. The computer-implemented method of claim 1, wherein the trained visual transformer model is trained on a plurality of unlabeled medical images using self-supervised learning.

11. The computer-implemented method of claim 1, wherein the status information of the one or more lesions is always stored in a database.

12. The computer-implemented method of claim 1, wherein the medical imaging data and the reference medical imaging data are not image registered with respect to each other.

13. A system for performing concurrent longitudinal lesion detection and analysis, the system comprising: a memory encoding a processor-executable routine; and A processor configured to access the memory and execute the processor-executable routine, wherein the processor-executable routine, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 12.

14. A non-transitory computer readable medium comprising processor executable code, which when executed by a processor causes the processor to perform the method according to any one of claims 1 to 12.

Images