Medical basis model for variable number of 3D anisotropic medical images

Abstract

Medical base model for variable number of 3D anisotropic medical images. Systems and methods for performing medical imaging analysis tasks using a base model are provided. One or more 3D (three-dimensional) medical images are received, each comprising a plurality of 2D (two-dimensional) slices. A first set of features is extracted from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network. Each respective feature in the first set of features is resampled based on a spatial location of one or more pixels of the respective 2D slice from which the respective feature was extracted. The resampled first set of features is encoded into a second set of features. A medical imaging analysis task is performed based on the second set of features. A result of the medical imaging analysis task is output.

Description

Technical Field

[0001] This invention generally relates to AI / ML (artificial intelligence / machine learning) based medical imaging analysis, and particularly to a medical foundation model for a variable number of 3D anisotropic medical images. Background Technology

[0002] Recent advances in base models have garnered significant attention and yielded notable progress across various applications. For instance, advancements in LLM (Large Language Models) have significantly improved question-answering tasks, while diffusion models have demonstrated excellent performance in image synthesis. However, the inherent complexity and heterogeneity of medical imaging data present significant challenges in creating base models.

[0003] Conventional basic models in the field of medical imaging typically accept a single image of fixed size. However, medical diagnosis often requires the use of multiple input medical images, such as multiple MR (magnetic resonance) images with different contrasts and varying anisotropic resolutions. Furthermore, the number of input images and the corresponding contrast acquired may vary across different diagnostic tasks and clinical settings. Summary of the Invention

[0004] According to one or more embodiments, a system and method are provided for performing a medical imaging analysis task using a base model. The system receives one or more 3D (three-dimensional) medical images, each comprising multiple 2D (two-dimensional) slices. A first feature set is extracted from the multiple 2D slices of the one or more 3D medical images using a machine learning-based encoding network. Each corresponding feature in the first feature set is resampled based on the spatial location of one or more pixels of the 2D slice from which the corresponding feature is extracted. The resampled first feature set is encoded into a second feature set. The medical imaging analysis task is performed based on the second feature set. The result of the medical imaging analysis task is output.

[0005] In one embodiment, the one or more 3D input medical images have at least one of different resolutions, different acquisition orientations, or different domains.

[0006] In one embodiment, each corresponding feature of the first feature set is resampled onto a 2D uniform pixel grid to form a 2D image in the feature space. The 2D images in the feature space are stacked to form one or more 3D isotropic images in the feature space.

[0007] In one embodiment, the spatial location of one or more pixels of a 2D slice includes the coordinates of the one or more pixels.

[0008] In one embodiment, a first feature set is encoded using location embedding and / or modality embedding.

[0009] In one embodiment, at least one of a transformer encoder or a state-space model is used to encode the resampled first feature set into a second feature set.

[0010] In one embodiment, at least one of a transformer encoder or a state-space model is used to perform a medical imaging analysis task.

[0011] These and other advantages of the invention will be apparent to those skilled in the art from the following detailed description and accompanying drawings. Attached Figure Description

[0012] Figure 1 A method for performing a medical imaging analysis task using a medical imaging fundamental model according to one or more embodiments is illustrated; Figure 2 A workflow for performing medical imaging analysis tasks using a medical imaging fundamental model according to one or more embodiments is illustrated; Figure 3 An exemplary artificial neural network that can be used to implement one or more embodiments is shown; Figure 4 This illustrates a convolutional neural network that can be used to implement one or more embodiments; Figure 5 A data flow diagram is shown that can be used to implement one or more embodiments of a generative adversarial network; Figure 6 A schematic structure is shown that can be used to implement a recurrent machine learning model of one or more embodiments; and Figure 7 A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. Detailed Implementation

[0013] This invention generally relates to methods and systems for performing medical imaging analysis tasks on a variable number of 3D anisotropic medical images using a medical fundamental model. Embodiments of the invention are described herein to provide an intuitive understanding of such methods and systems. Digital images often consist of digital representations of one or more objects (or shapes). In this document, the digital representation of an object is often described in terms of identifying and manipulating the object. Such manipulation is a virtual manipulation performed in the memory or other circuitry / hardware of a computer system. Therefore, it should be understood that embodiments of the invention can be performed within a computer system using data stored within the computer system. Furthermore, references to pixels of an image herein can similarly refer to voxels of the image, and vice versa.

[0014] The embodiments described herein provide a novel foundational model framework that efficiently handles heterogeneous input medical images with varying anisotropic resolutions, acquisition orientations, and / or numbers of different domains. This framework eliminates the need to train multiple models or encoders to accommodate varying numbers of input medical images and removes the requirement for extensive upsampling / downsampling of input medical images with different anisotropic resolutions. The foundational model according to the embodiments described herein can utilize all available input data with minimal preprocessing, and the pre-trained foundational model can be used to solve various downstream tasks involving input medical images with varying requirements and numbers of variations.

[0015] Figure 1 A method 100 for performing a medical imaging analysis task using a medical imaging fundamental model, according to one or more embodiments, is illustrated. The steps and sub-steps of method 100 can be performed by one or more suitable computing devices, such as… Figure 7 Computer 702. Figure 2 A workflow 200 for performing medical imaging analysis tasks using a medical imaging basic model, according to one or more embodiments, is shown. Figure 1 and Figure 2 They will be described together.

[0016] exist Figure 1 At step 102, one or more 3D (three-dimensional) medical images are received, each comprising multiple 2D (two-dimensional) slices. In one example, such as Figure 2 As shown in workflow 200, the one or more 3D medical images are 3D medical images 202-A, 202-B, ..., 202-N (collectively referred to as 3D medical images 202). 3D medical image 202 comprises 32 2D slices, each slice having a resolution of 256 × 256 pixels. The one or more 3D medical images can be anisotropic medical images depicting any anatomical object of interest to the patient.

[0017] In one embodiment, the one or more 3D medical images have different resolutions, acquisition orientations, and / or domains. An acquisition orientation refers to a different plane or angle from which one or more 3D medical images are acquired to observe various cross-sections of a patient's body. Examples of acquisition orientations include axial, coronal, and sagittal orientations. As used herein, a domain of a medical image refers to the modality of the medical image and the protocol used to acquire the medical image in that modality. The modality of the one or more input medical images may include, for example, MRI (Magnetic Resonance Imaging), CT (Computed Tomography), US (Ultrasound), X-ray, SPECT (Single Photon Emission Tomography), PET (Positron Emission Tomography), or any other medical imaging modality or combination of medical imaging modalities. The protocol used to acquire the medical image may include, for example, acquisition sequences or techniques for acquiring the medical image, such as, for example, T1-weighted, T2-weighted, proton density-weighted MRI images, contrast and non-contrast images, CT images captured using low kV and high kV, or low and high resolution medical images. Therefore, the domain can be entirely different medical imaging modalities or different image protocols within the same overall imaging modality. The one or more input medical images can be represented in an image space (e.g., as pixel or voxel values ​​in spatial coordinates) or a latent space (e.g., as a low-dimensional compressed representation of one or more medical images represented as feature vectors).

[0018] For example, this can be achieved by capturing one or more 3D medical images from an image acquisition device (e.g., Figure 7 The image acquisition device 714) directly receives one or more 3D medical images, and transmits them from the storage device or memory of the computer system (e.g., ...). Figure 7 The computer 702 loads one or more 3D medical images from its storage device 712 or memory 710, or from a remote computer system (e.g., Figure 7 The computer (702) receives one or more 3D medical images. Such a computer system or remote computer system may include one or more patient databases, such as, for example, EHR (Electronic Health Record), EMR (Electronic Medical Record), PHR (Personal Health Record), HIS (Health Information System), RIS (Radiology Information System), PACS (Picture Archiving and Communication System), LIMS (Laboratory Information Management System), or any other suitable database or system.

[0019] exist Figure 1 At step 104, a first feature set is extracted from multiple 2D slices of the one or more 3D medical images using a machine learning-based encoding network.

[0020] In one embodiment, the machine learning-based encoding network is a 2D CNN (Convolutional Neural Network), such as, for example, 2D UNet. For example, as... Figure 2 As shown in workflow 200, features 206-A, 206-B, ..., 206-N (collectively referred to as features 206) are extracted from 3D medical images 202-A, 202-B, ..., 202-N by a 2D CNN 204. However, the machine learning-based encoding network can be implemented according to any other suitable machine learning-based architecture. The machine learning-based encoding network receives each of one or more 2D slices of the 3D medical images as input and generates a first feature set as output. The features of the first feature set are low-dimensional compressed representations of the 2D slices represented as feature vectors.

[0021] Machine learning-based encoding networks downsample 2D slices by progressively reducing spatial resolution, while capturing and encoding the most important features in a compressed, low-dimensional representation. For example, ... Figure 2 As shown in workflow 200, 2D CNN 204 downsamples 2D slices of 3D medical image 202 to a resolution of 32×32×32 in a feature space of a total of 765 features.

[0022] exist Figure 1 At step 106, each corresponding feature of the first feature set is resampled based on the spatial location of one or more pixels of the 2D slice from which the corresponding feature is extracted. In one example, such as Figure 2 As shown in workflow 200, feature 206 is resampled at feature map resampling block 208 to generate resampled feature 210.

[0023] Features are resampled to align the spatial locations of one or more pixels in a 2D slice from which features are extracted. Features are resampled onto a uniform 2D pixel grid to form 2D slices in the feature space of one or more 3D medical images. The 2D slices / images in the feature space are stacked to form an isotropic 3D image in the feature space. For example, in... Figure 2 In workflow 200, features 206 are resampled on a 2D uniform pixel grid to form 32 2D slices in image space, and these 2D slices are stacked to form a 3D isotropic image in feature space. Therefore, resampling provides an isotropic 3D image in the feature space of one or more anisotropic 3D medical images in image space. The first set of resampled features generated from different anisotropic images by a machine learning-based encoding network results in a more similar and isotropic resolution because high-resolution dimensions are extracted as high-level features with lower resolution. Resampling does not lead to significant information loss because the level of anisotropy is reduced when the 2D slices are downsampled.

[0024] In one embodiment, the spatial location includes the coordinates of each pixel of a 2D slice of one or more 3D medical images (e.g., in the coordinate system of the image acquisition device). The coordinates may be generated by the image acquisition device at the acquisition point and stored in the header of one or more 3D medical images.

[0025] In one embodiment, resampled features are encoded using localization embeddings (e.g., sinusoidal or learnable) based on the spatial location of one or more pixels of a 2D slice from which features are extracted. Thus, features located at the same relative position within the 3D medical image but from different 3D medical images will have the same localization embedding. Localization embeddings help preserve the spatial relationships of features. In one embodiment, additionally or alternatively, modal embeddings are used to encode resampled features. Localization embeddings and / or modal embeddings can be fixed or trainable. Modal embeddings can also be derived from a domain (e.g., modal or contrastive name) using a language encoder.

[0026] exist Figure 1 At step 108, the resampled first feature set is encoded into a second feature set. In one embodiment, a machine learning-based encoder network, such as a transformer encoder, is used to encode the resampled first feature set. In another embodiment, a state-space model is used to encode the resampled first feature set. For example, as... Figure 2 As shown in workflow 200, the converter encoder / state space model 212 encodes feature 210 to generate additional features ( Figure 2 (Not shown in the image). However, any other suitable method can be used to encode the resampled first feature set into the second feature set. A machine learning-based encoder network / state-space model receives the resampled first feature set as input and generates the second feature set as output.

[0027] exist Figure 1 At step 110, a medical imaging analysis task is performed based on the second feature set. In one embodiment, one or more machine learning-based task networks, such as task-specific decoder heads (e.g., transformer decoders), can be used to perform the medical imaging analysis task. The machine learning-based task network receives the second feature set as input and generates the medical imaging analysis task as output. In another embodiment, a state-space model can be used to perform the medical imaging analysis task.

[0028] In one embodiment, the medical imaging analysis task is image synthesis for generating a synthetic image from one or more medical images. The synthetic image may be in a domain, acquisition orientation, or resolution not present in the one or more 3D medical images. The medical imaging analysis task may additionally or alternatively include any other suitable tasks, such as, for example, detection, segmentation, classification, quantization, etc.

[0029] exist Figure 1 At step 114, the results of the medical imaging analysis task are output. For example, this can be done via a display device on a computer system (e.g., [display device]). Figure 7 The results are displayed on the I / O 708 of the computer 702, and stored in the computer system's memory or storage device (e.g., Figure 7 The results are stored on the memory 710 or storage device 712 of the computer 702, or by transmitting the results to a remote computer system (e.g., Figure 7 The computer (702) is used to output the results of medical imaging analysis tasks.

[0030] Advantageously, the embodiments described herein address computational cost challenges by implementing self-attention masks for the transformer or by employing a state-space model such as, for example, Mamba. Assuming the underlying model according to the embodiments described herein is adapted to multiple input medical images, the token length can be considerable. Utilizing a state-space model can offer advantages in significantly reducing computational costs. Alternatively, in the case where a transformer architecture is employed, masked self-attention can be applied. Specifically, self-attention computation occurs across all tokens within two specific dimensions for each input medical image, and across all tokens in the same single dimension across all images. This approach effectively optimizes computational efficiency while maintaining the model's ability to capture relevant interactions among the tokens.

[0031] During the pre-training phase, self-supervised learning employing a masked autoencoder strategy can be used. Subsequently, for the downstream task (i.e., for performing medical imaging analysis), the transformer or state-space model is connected to a specific decoder tailored for each task type, such as a classification head or a segmentation head. To enhance training robustness, conventional augmentation techniques, such as randomly dropping input modalities, can be implemented. Additionally, random sampling of a subset of the input medical images or the total tokens for each case / subject is utilized to alleviate GPU memory constraints, thereby optimizing computational resources.

[0032] In one example, the one or more machine learning-based task networks include a linear decoder. For example, CMR (cardiac MR) is the gold standard for estimating ejection fraction, while echocardiography (or echocardiography, echo) provides real-time imaging as well as live quantification and analysis. These two modalities are often acquired in different orientations. By combining these two modalities, diagnostic accuracy can be improved, particularly in detecting cardiac abnormalities. In this scenario, input 3D medical image modalities, such as, for example, CMR and echocardiography, are received at step 102, and the linear decoder outputs at step 110, generating a score indicating whether cardiac activity is normal or abnormal.

[0033] Beyond its diagnostic applications, the embodiments described herein can also be employed during interventional procedures. In one example, the one or more machine learning-based task networks include a multi-task decoder. For instance, during LAA (left atrial appendage) closure procedures, both cardiac CT and live intracardiac echocardiography are often used to estimate the LAA ostium size, which determines the appropriate implantation device. In this case, two decoders can be utilized, each translating a feature map into a segmentation map of the LAA ostium in its corresponding modality. This system acts as a registration engine, ensuring consistent anatomical segmentation across different modalities. This allows physicians to accurately assess anatomical changes by comparing preoperative CT images with live echocardiographic images.

[0034] The embodiments described herein relate to a claimed system and a claimed method. Features, advantages, or alternative embodiments described herein may be assigned to other claims, and vice versa. In other words, the system claims and embodiments may be improved using features described or claimed in the context of the corresponding method. In this case, the functional features of the method are implemented by the physical units of the system.

[0035] Furthermore, the specific embodiments described herein are relating to methods and systems for utilizing trained machine learning models, and to methods and systems for providing trained machine learning models. Features, advantages, or alternative embodiments described herein may be assigned to other claimed objects, and vice versa. In other words, the claims and embodiments for providing trained machine learning models may be modified using features described or claimed in the context of utilizing trained machine learning models, and vice versa. In particular, the dataset used in the methods and systems for utilizing trained machine learning models may have the same properties and characteristics as the corresponding dataset used in the methods and systems for providing trained machine learning models, and the trained machine learning model provided by the corresponding method and system may be used in the methods and systems for utilizing trained machine learning models.

[0036] Generally speaking, trained machine learning models mimic human cognitive functions associated with other human thought processes. Specifically, through training on training data, machine learning models can adapt to new situations and detect and extrapolate patterns. Another term for a “trained machine learning model” is a “trained function.”

[0037] Generally, the parameters of a machine learning model can be adapted through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Further, representation learning (an alternative term is "feature learning") can be used. Specifically, the parameters of a machine learning model can be iteratively adapted through several training steps. Specifically, within training, a specific cost function can be minimized. Specifically, within the training of a neural network, the backpropagation algorithm can be used.

[0038] In particular, machine learning models, such as machine learning-based encoding networks utilized, for example, in steps 104 or 108, or in Figure 1 Step 110 utilizes a machine learning-based decoding network, or Figure 2 The 2D CNN 204 or transformer encoder / state space model 212 may include, for example, neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or machine learning models may be based on, for example, k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network may be, for example, a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be, for example, an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0039] Figure 3 An embodiment of an artificial neural network 300 is shown, which can be used to implement one or more machine learning models described herein. Alternative terms for “artificial neural network” are “neural network,” “artificial neural network,” or “neural network.”

[0040] The artificial neural network 300 includes nodes 320, ..., 332 and edges 340, ..., 342, where each edge 340, ..., 342 is a directed connection from a first node 320, ..., 332 to a second node 320, ..., 332. Generally, the first nodes 320, ..., 332 and the second nodes 320, ..., 332 are different nodes 320, ..., 332; it is also possible that the first nodes 320, ..., 332 and the second nodes 320, ..., 332 are the same. For example, in... Figure 3In the diagram, edge 340 is a directed connection from node 320 to node 323, and edge 342 is a directed connection from node 330 to node 332. Edges 340, ..., 342 from the first node 320, ..., 332 to the second node 320, ..., 332 are also labeled as "incoming edges" of the second node 320, ..., 332 and "outgoing edges" of the first node 320, ..., 332.

[0041] In this embodiment, nodes 320, ..., 332 of the artificial neural network 300 can be arranged in layers 310, ..., 313, wherein the layers may include an inherent order introduced by edges 340, ..., 342 between nodes 320, ..., 332. Specifically, edges 340, ..., 342 may exist only between adjacent node layers. In the illustrated embodiment, an input layer 310 includes only nodes 320, ..., 322 without any incoming edges, an output layer 313 includes only nodes 331, 332 without any outgoing edges, and hidden layers 311, 312 are located between the input layer 310 and the output layer 313. Generally, the number of hidden layers 311, 312 can be arbitrarily chosen. The number of nodes 320, ..., 322 in the input layer 310 is typically related to the number of input values ​​of the neural network, and the number of nodes 331, 332 in the output layer 313 is typically related to the number of output values ​​of the neural network.

[0042] Specifically, (real) numbers can be assigned as values ​​to each node 320, ..., 332 of the neural network 300. Here, The values ​​of the i-th nodes 320, ..., 332 in the n-th layers 310, ..., 313 are labeled. The values ​​of nodes 320, ..., 322 in the input layer 310 are equivalent to the input values ​​of the neural network 300, and the values ​​of nodes 331, 332 in the output layer 313 are equivalent to the output values ​​of the neural network 300. Furthermore, each edge 340, ..., 342 may include a weight as a real number, specifically a real number within the interval [-1, 1] or the interval [0, 1]. Here, This indicates the weight of the edge between the i-th node (320, ..., 332) of layer m (310, ..., 313) and the j-th node (320, ..., 332) of layer n (310, ..., 313). Further, abbreviated... Targeted weights To define.

[0043] Specifically, to calculate the output value of neural network 300, the input values ​​are propagated through the neural network. Specifically, the values ​​of nodes 320, ..., 332 in the (n+1)th layer 310, ..., 313 can be calculated based on the values ​​of nodes 320, ..., 332 in the nth layer 310, ..., 313 using the following formula: .

[0044] In this paper, the function f is the transfer function (another term is the "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logic functions, generalized logic functions, hyperbolic tangent functions, arctangent functions, error functions, smooth step functions), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0045] Specifically, the value is propagated layer by layer through the neural network, wherein the value of the input layer 310 is given by the input of the neural network 300, wherein the value of the first hidden layer 311 can be calculated based on the value of the input layer 310 of the neural network, wherein the value of the second hidden layer 312 can be calculated based on the value of the first hidden layer 311, and so on.

[0046] To set the value of the edge Training data is required to train the neural network 300. Specifically, the training data includes training input data and training output data (denoted as t). i For the training step, neural network 300 is applied to the training input data to generate the computed output data. Specifically, the training data and the computed output data include multiple values, the number of which is equal to the number of nodes in the output layer.

[0047] Specifically, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network (backpropagation algorithm). Specifically, the weights change according to the following formula: in It is the learning rate, and if the (n+1)th layer is not an output layer, and , If the (n+1)th layer is an output layer 313, where f' is the first derivative of the activation function, and It is the comparison training value of the j-th node in output layer 313. Then number Based on Recursively calculate as .

[0048] A convolutional neural network (CNN) is a neural network that uses convolution operations instead of general matrix multiplication in at least one of its layers (so-called "convolutional layers"). Specifically, a convolutional layer performs a dot product of one or more convolutional kernels with the input data / image of the convolutional layer, wherein the entries for said one or more convolutional kernels are parameters or weights adapted through training. In particular, one can use Frobenius inner products and ReLU activation functions. A CNN may include additional layers such as pooling layers, fully connected layers, and normalization layers.

[0049] By using convolutional neural networks, input images can be processed very efficiently because convolution operations based on different kernels can extract various image features. This allows relevant image features to be found during training by adapting the weights of the convolutional kernels. Furthermore, due to weight sharing within the convolutional kernels, fewer parameters need to be trained, preventing overfitting during training and allowing for faster training or more layers in the network, thus improving network performance.

[0050] Figure 4 An embodiment of a convolutional neural network 400 is illustrated, which can be used to implement one or more machine learning models described herein. In the illustrated embodiment, the convolutional neural network 400 includes an input node layer 410, a convolutional layer 411, a pooling layer 413, a fully connected layer 414, and an output node layer 416, as well as hidden node layers 412 and 414. Alternatively, the convolutional neural network 400 may include a plurality of convolutional layers 411, a plurality of pooling layers 413, and a plurality of fully connected layers 415, as well as other types of layers. The order of the layers can be arbitrarily chosen, but typically the fully connected layer 415 is used as the last layer before the output layer 416.

[0051] Specifically, within the convolutional neural network 400, nodes 420, 422, and 424 of node layers 410, 412, and 414 can be considered as arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of nodes 420, 422, and 424 in the nth node layer 410, 412, and 414, indexed by i and j, can be denoted as x(n)[i, j]. However, the arrangement of nodes 420, 422, and 424 in a node layer 410, 412, and 414 in this way has no effect on the computations performed within the convolutional neural network 400, because these are given only by the structure and weights of the edges.

[0052] Convolutional layer 411 is a connecting layer between the preceding node layer 410 (with node value x(n-1)) and the following node layer 412 (with node value x(n)). Specifically, convolutional layer 411 is characterized by the structure and weights of the input edges that form the convolution operation based on a specific number of kernels. In particular, the structure and weights of the edges of convolutional layer 411 are selected such that the value x(n) of node 422 of the following node layer 412 is computed as a convolution based on the value x(n-1) of node 420 of the preceding node layer 410. Convolution In the two-dimensional case, it is defined as .

[0053] Here, the kernel K is a d-dimensional matrix (in this embodiment, a two-dimensional matrix), which is typically small compared to the number of nodes 420, 422 (e.g., a 3×3 or 5×5 matrix). Specifically, this implies that the weights of the edges in convolutional layer 411 are not independent, but are chosen such that they produce the convolution equation. In particular, for a 3×3 kernel, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight), independent of the number of nodes 420, 422 in the preceding node layer 410 and the following node layer 412.

[0054] Generally, convolutional neural networks 400 use node layers 410, 412, and 414 with multiple channels, especially since multiple kernels are used in convolutional layer 411. In those cases, the node layer can be considered as a (d+1)-dimensional matrix (the first dimension indexing the channels). The operation of convolutional layer 411 is then defined as a two-dimensional example: in Corresponding to the a-th channel of the previous node layer 410, Corresponding to the b-th channel of the subsequent node layer 412, and This corresponds to one of the kernels. If convolutional layer 411 acts on a preceding node layer 410 with A channels and outputs a following node layer 412 with B channels, then there exists... An independent d-dimensional kernel .

[0055] Generally, activation functions are used in convolutional neural networks 400. In this embodiment, ReLU (an abbreviation for "Rectified Linear Unit") is used, where... This makes the action of convolutional layer 411 in the two-dimensional example: .

[0056] It is also possible to use other activation functions, such as ELU (an abbreviation for "Exponential Linear Unit"), LeakyReLU, Sigmoid, Tanh, or Softmax.

[0057] In the shown embodiment, the input layer 410 includes 36 nodes 420 arranged in a two-dimensional 6×6 matrix. The first hidden node layer 412 includes 72 nodes 422 arranged in two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with a 3×3 kernel within the convolutional layer 411. Equivalently, the nodes 422 of the first hidden node layer 412 can be interpreted as being arranged in a three-dimensional 2×6×6 matrix, where the first dimension corresponds to the channel dimension.

[0058] The advantage of using convolutional layer 411 is that it can take advantage of the spatial local correlation of the input data by implementing local connectivity patterns between nodes in neighboring layers, in particular by having each node connected to a small region of nodes in the previous layer.

[0059] Pooling layer 413 is the connecting layer between the previous node layer 412 (with node value x(n-1)) and the next node layer 414 (with node value x(n)). Specifically, pooling layer 413 can be characterized by the structure and weights of the edges and activation functions that form the pooling operation based on the nonlinear pooling function f. For example, in the two-dimensional case, the value x(n) of node 424 of the next node layer 414 can be calculated based on the value x(n-1) of node 422 of the previous node layer 412 as follows: .

[0060] In other words, by using pooling layer 413, the number of nodes 422 and 424 can be reduced. This is achieved by reducing the number of nodes. The adjacent nodes 422 are repositioned in the previous node layer 412, and the calculation is performed using the individual nodes 422 in the subsequent node layer 414 as a function of the values ​​of the number of adjacent nodes. Specifically, the pooling function f can be a maximum function, an average function, or an L2 norm function. Specifically, for the pooling layer 413, the weights of the incoming edges are fixed and are not modified through training.

[0061] The advantage of using pooling layer 413 is that it reduces the number of nodes 422 and 424 and the number of parameters. This results in a reduction in the computational cost of the network and controls overfitting.

[0062] In the illustrated embodiment, pooling layer 413 is a max-pooling layer that replaces four adjacent nodes with only one node, the maximum of the values ​​of the four adjacent nodes. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0063] Generally, the last few layers of a convolutional neural network 400 are fully connected layers 415. Fully connected layer 415 is the connection layer between the preceding node layer 414 and the following node layer 416. A characteristic of fully connected layer 413 is that most, in particular all, of the edges exist between nodes 414 of the preceding node layer 414 and nodes 416 of the following node layer, and the weights of each of these edges can be adjusted individually.

[0064] In this embodiment, the nodes 424 of the preceding node layer 414 of the fully connected layer 415 are displayed as a two-dimensional matrix, and additionally as unrelated nodes (indicated as node rows, where the number of nodes is reduced for better presentation). This operation is also labeled "flattening". In this embodiment, the number of nodes 426 in the following node layer 416 of the fully connected layer 415 is less than the number of nodes 424 in the preceding node layer 414. Alternatively, the number of nodes 426 can be equal to or greater than the number of nodes 424.

[0065] Furthermore, in this embodiment, a Softmax activation function is used within the fully connected layer 415. By applying the Softmax function, the sum of the values ​​of all nodes 426 in the output layer 416 is 1, and all values ​​of all nodes 426 in the output layer 416 are real numbers between 0 and 1. In particular, if the convolutional neural network 400 is used to classify input data, the values ​​of the output layer 416 can be interpreted as the probability that the input data falls into one of the different categories.

[0066] Specifically, the convolutional neural network 400 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as dropping nodes 420, ..., 424, random pooling, the use of artificial data, and weight decay based on L1 or L2 norm or maximum norm constraints.

[0067] According to one aspect, a machine learning model may include one or more Residual Networks (ResNets). Specifically, a ResNet is an artificial neural network that includes at least one skip connection for skipping at least one layer of the artificial neural network. Specifically, a ResNet may be a convolutional neural network that includes one or more skip connections that skip one or more convolutional layers respectively. According to some examples, a ResNet can be represented as an m-layer ResNet, where m is the number of layers in the corresponding architecture, and according to some examples, it can take values ​​of 34, 50, 101, or 152. According to some examples, such an m-layer ResNet may correspondingly include (m-2) / 2 skip connections.

[0068] Skipping connections can be viewed as bypassing, which feeds the output of a previous layer directly to a layer following one or more bypassed layers. Instead of directly fitting the desired mapping, the bypassed layer then has to fit a residual mapping that "balances" the output of the directly fed-in layer.

[0069] Fitting residual mappings is computationally easier to optimize than directional mappings. More importantly, this alleviates the vanishing / exploding gradient problem during optimization when training machine learning models: if bypassed layers encounter such a problem, their contribution can be skipped by regularizing the output of the direct feedback. Therefore, one advantage of using ResNet is that much deeper networks can be trained.

[0070] Generative adversarial models (GA models) consist of a generator function and a discriminator function. The generator function creates synthetic data, and the discriminator function distinguishes between synthetic and real data. By training the generator and / or discriminator functions, the generator function is configured to create synthetic data that is incorrectly classified as real by the discriminator function, while the discriminator function is configured to distinguish between real data and synthetic data generated by the generator function. In game theory, GA models can be interpreted as zero-sum games. The training of the generator and / or discriminator functions is specifically based on minimizing a cost function.

[0071] By using a GA (Generative Approach) model, synthetic data with the same characteristics as the training dataset can be generated based on the training dataset. Training a GA model can be based on unannotated data (unsupervised learning), resulting in low workload when training the GA model.

[0072] Figure 5 A data flow diagram is shown according to one or more embodiments of an embodiment for using a generative adversarial network (GAN) to create synthetic output data G(x) 508 based on input data x 502 that is indistinguishable from real output data y 504. The synthetic output data G(x) 508 has the same structure as the real output data y 504, but its content is not derived from real-world data.

[0073] The Generative Adversarial Network (GAN) consists of a jointly trained generator function G 506 and a classifier function C 510. The generator function G 506 is tasked with providing realistic synthetic output data G(x) 508 based on the input data x 502, and the classifier function C 510 is tasked with distinguishing between the real output data y 504 and the synthetic output data G(x) 508. Specifically, the output of the classifier function C 510 is a real number between 0 and 1 corresponding to the probability that the input value is real data, such that an ideal classifier function would compute the output value of the real data y 504. and the output value of the synthetic data G(x) 508 .

[0074] During training, the parameters of the generator function G 506 are adapted such that the synthetic output data G(x) 508 has the same characteristics as the real output data y 504, making it impossible for the classifier function C 510 to distinguish between the real and synthetic data. Simultaneously, the parameters of the classifier function C 510 are adapted so that it distinguishes between the real and synthetic data in the best possible way. Here, training relies on pairs including input data x 502 and corresponding real output data y 504. Within a single training step, the generator function G 506 is applied to the input data x 502 to generate the synthetic output data G(x) 508. Further, the classifier function C 510 is applied to the real output data y 504 to generate the first classification result C(y) 514. Additionally, the classifier function C 510 is applied to the synthetic output data G(x) 508 to generate the second classification result C(G(x)) 512.

[0075] The parameters of the adaptation generation function G 506 and the classifier function C 510 are respectively based on minimizing the cost function using the backpropagation algorithm. In this embodiment, the cost function K of the classifier function C 510 is... C yes The BCE designation is defined as follows: The binary cross-entropy. By using this cost function, the true output data is incorrectly classified as synthetic (made by...). (indication) and incorrectly classifying synthetic output data as real (indication as) Both of these increase the cost function K to be minimized. C Furthermore, the cost function K of the generator function G506... G yes By using this cost function, the correctly classified synthetic output data (indicated as...) This leads to the cost function K G The increase is minimized.

[0076] Specifically, a recurrent machine learning model is a machine learning model whose output depends not only on the input values ​​and the parameters of the machine learning model adapted by the training process, but also on the hidden state vector, which is based on previous inputs used for the recurrent machine learning model. In particular, a recurrent machine learning model may include additional stored states or additional structures incorporating time delays or feedback loops.

[0077] Specifically, the underlying structure of a recurrent machine learning model can be a neural network, which can be labeled as a recurrent neural network. Such a recurrent neural network can be described as an artificial neural network, where the connections between nodes form a directed graph along a time series. In particular, a recurrent neural network can be interpreted as a directed acyclic graph. Specifically, a recurrent neural network can be a finite-spiking recurrent neural network or an infinite-spiking recurrent neural network (where a finite-spiking network can be unfolded and replaced with a strictly feedforward neural network, and an infinite-spiking network cannot be unfolded and replaced with a strictly feedforward neural network).

[0078] Specifically, training recurrent neural networks can be based on the BPTT algorithm (an acronym for "backpropagation based on time"), the RTRL algorithm (an acronym for "real-time recursive learning"), and / or a genetic algorithm.

[0079] By using recurrent machine learning models, input data that includes sequences of variable length can be used. Specifically, this implies that the method can be used not only with a fixed number of input datasets (and requires different training for each additional number of input datasets used as input), but also with any number of input datasets. This implies that, independent of the number of input datasets contained in different sequences, the entire training dataset can be used within training, and the training data is not reduced to training data corresponding to a specific number of successive input datasets.

[0080] Figure 6 The loop representation 602 and the expanded representation 604 illustrate a schematic structure of a recurrent machine learning model F that can be used to implement one or more machine learning models described in this paper. The recurrent machine learning model takes several input datasets x, x1, …, x2. N Take 606 as input and create a corresponding set y, y1, …,y of the output dataset. N 608. Furthermore, the output depends on the so-called hidden vectors h, h1, …, h N 610, which implicitly includes information about the input dataset previously used as input to the recurrent machine learning model F 612. This is achieved by using these hidden vectors h, h1, ..., h N 610, can take advantage of the order of the input dataset.

[0081] In a single step of processing, the recurrent machine learning model F612 will process the hidden vector h created in the previous step. n-1 and input dataset x n Take it as input. Within this step, the recurrent machine learning model F generates the updated hidden vector h. n and output dataset y nAs output. In other words, a processing step calculates... Alternatively, by splitting the recurrent machine learning model F612 into a part F(y) that computes the output data and a part F(h) that computes the hidden vectors, one processing step computes... and For the first processing step, h0 can be randomly selected, or it can be filled using all entries that are zero. The parameters of the recurrent machine learning model F 612, which was previously trained on the training dataset, remain unchanged between different processing steps.

[0082] In particular, the output data and hidden vectors of the processing step depend on all the previous input datasets used in the previous steps. and .

[0083] The systems, apparatus, and methods described herein can be implemented using digital circuitry or using one or more computers employing known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0084] The systems, apparatus, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0085] The systems, apparatuses, and methods described herein can be implemented in a network-based cloud computing system. In such a system, a server or other processor connected to the network communicates with one or more client computers via the network. For example, a client computer may communicate with the server via a web browser application residing on and operating on the client computer. The client computer may store data on the server and access the data via the network. The client computer may transmit requests for data or online services to the server via the network. The server may perform the requested service and provide data to one or more client computers. The server may also transmit data adapted to cause the client computer to perform specified functions, such as performing calculations, displaying specified data on a screen, etc. For example, the server may transmit requests adapted to cause the client computer to perform one or more steps or functions of the methods and workflows described herein, including... Figure 1 or Figure 2 One or more of the steps or functions. Specific steps or functions of the methods and workflows described herein, including... Figure 1 or Figure 2 One or more of the steps or functions described herein may be executed by a server or another processor in a web-based cloud computing system. Specific steps or functions of the methods and workflows described herein include... Figure 1 or Figure 2 One or more of the steps can be performed by a client computer in a web-based cloud computing system. The steps or functions of the methods and workflows described herein include... Figure 1 or Figure 2 One or more of the steps can be performed by servers and / or client computers in a web-based cloud computing system in any combination.

[0086] The systems, apparatuses, and methods described herein can be implemented using a computer program product tangibly embodied in an information carrier, such as a non-transitory machine-readable storage device, for execution by a programmable processor; and the methods and workflow steps described herein include Figure 1 or Figure 2 One or more of the steps or functions can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a specific activity or produce a specific result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0087] Figure 7 A high-level block diagram of an example computer 702, which can be used to implement the systems, apparatus, and methods described herein, is depicted. Computer 702 includes a processor 704 operatively coupled to a data storage device 712 and a memory 710. Processor 704 controls the overall operation of computer 702 by executing computer program instructions that define such operations. The computer program instructions may be stored in the data storage device 712 or other computer-readable medium and loaded into memory 710 when execution of the computer program instructions is desired. Therefore, Figure 1 or Figure 2 The methods and workflow steps or functions can be defined by computer program instructions stored in memory 710 and / or data storage device 712, and controlled by processor 704 that executes the computer program instructions. For example, the computer program instructions can be implemented as computer-executable code programmed by those skilled in the art to perform... Figure 1 or Figure 2 The methods and workflow steps or functions. Therefore, by executing computer program instructions, processor 704 performs... Figure 1 or Figure 2 The computer 702 may include methods and workflow steps or functions. The computer 702 may also include one or more network interfaces 706 for communicating with other devices via a network. The computer 702 may also include one or more input / output devices 708 (e.g., a display, keyboard, mouse, speaker, buttons, etc.) that enable a user to interact with the computer 702.

[0088] Processor 704 may include both general-purpose and special-purpose microprocessors, and may be the sole processor or one of multiple processors in computer 702. For example, processor 704 may include one or more central processing units (CPUs). Processor 704, data storage device 712, and / or memory 710 may include one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs), or may be supplemented or incorporated therein by one or more ASICs and / or one or more FPGAs.

[0089] Each of the data storage device 712 and the memory 710 includes a tangible, non-transitory computer-readable storage medium. Each of the data storage device 712 and the memory 710 may include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital universal disc read-only memory (DVD-ROM), or other non-volatile solid-state storage devices.

[0090] Input / output device 708 may include peripheral devices such as printers, scanners, displays, etc. For example, input / output device 708 may include display devices such as cathode ray tube (CRT) or liquid crystal display (LCD) monitors for displaying information to a user, keyboards, and pointing devices such as mice or trackballs through which the user can provide input to computer 702.

[0091] Image acquisition device 714 can be connected to computer 702 to input image data (e.g., medical images) into computer 702. It is possible that image acquisition device 714 and computer 702 are implemented as a single device. It is also possible that image acquisition device 714 and computer 702 communicate wirelessly via a network. In a possible embodiment, computer 702 may be located remotely relative to image acquisition device 714.

[0092] Any or all of the systems, apparatuses and methods discussed herein can be implemented using one or more computers, such as computer 702.

[0093] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and Figure 7 It is a high-level representation of some of the components of such a computer for illustrative purposes.

[0094] In this patent application, nouns and pronouns referring to people generally do not specify a particular gender.

[0095] The foregoing detailed description should be understood as illustrative and exemplary in every respect, not restrictive, and the scope of the invention disclosed herein should not be determined from the detailed description, but rather from the claims as interpreted in the full breadth permitted by patent law. It should be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention. Various other combinations of features can be implemented by those skilled in the art without departing from the scope and spirit of the invention.

[0096] The following is a list of non-limiting illustrative embodiments disclosed herein: Illustrative Example 1. A computer-implemented method comprising: receiving one or more 3D (three-dimensional) medical images, each comprising a plurality of 2D (two-dimensional) slices; extracting a first feature set from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network; resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of the 2D slice from which the corresponding feature was extracted; encoding the resampled first feature set into a second feature set; performing a medical imaging analysis task based on the second feature set; and outputting the result of the medical imaging analysis task.

[0097] Illustrative Example 2. The computer-implemented method according to Illustrative Example 1, wherein the one or more 3D medical images have at least one of different resolutions, different acquisition orientations, or different domains.

[0098] Illustrative Example 3. A computer-implemented method according to any one of Illustrative Examples 1-2, wherein resampling each corresponding feature of a first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: resampling each corresponding feature of the first feature set onto a 2D uniform pixel grid to form a 2D image in feature space.

[0099] Illustrative Example 4. The computer-implemented method according to Illustrative Example 3, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted further comprises: stacking 2D images in a feature space to form one or more 3D isotropic images in the feature space.

[0100] Illustrative Example 5. A computer-implemented method according to any one of Illustrative Examples 1-4, wherein the spatial location of one or more pixels of a 2D slice includes the coordinates of one or more pixels.

[0101] Illustrative Example 6. A computer-implemented method according to any one of Illustrative Examples 1-5, wherein resampling each corresponding feature of a first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: encoding the first feature set using localization embedding.

[0102] Illustrative Example 7. A computer-implemented method according to any one of Illustrative Examples 1-6, wherein resampling each corresponding feature of a first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: encoding the first feature set using modal embedding.

[0103] Illustrative Example 8. A computer-implemented method according to any one of Illustrative Examples 1-7, wherein encoding a resampled first feature set into a second feature set comprises: encoding the resampled first feature set into a second feature set using at least one of a transformer encoder or a state-space model.

[0104] Illustrative Example 9. A computer-implemented method according to any one of Illustrative Examples 1-8, wherein performing a medical imaging analysis task based on a second feature set includes: performing the medical imaging analysis task using at least one of a transformer encoder or a state-space model.

[0105] Illustrative Example 10. An apparatus comprising: means for receiving one or more 3D (three-dimensional) medical images, each of the one or more 3D (three-dimensional) medical images comprising a plurality of 2D (two-dimensional) slices; means for extracting a first feature set from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network; means for resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of the 2D slice from which the corresponding feature was extracted; means for encoding the resampled first feature set into a second feature set; means for performing a medical imaging analysis task based on the second feature set; and means for outputting the result of the medical imaging analysis task.

[0106] Illustrative Example 11. The apparatus according to Illustrative Example 10, wherein the one or more 3D medical images have at least one of different resolutions, different acquisition orientations, or different domains.

[0107] Illustrative Example 12. The apparatus according to any one of Illustrative Examples 10-11, wherein the component for resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted includes: a component for resampling each corresponding feature of the first feature set onto a 2D uniform pixel grid to form a 2D image in the feature space.

[0108] Illustrative Example 13. The apparatus according to any one of Illustrative Examples 10-12, wherein the component for resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted further includes: a component for stacking 2D images in the feature space to form one or more 3D isotropic images in the feature space.

[0109] Illustrative Example 14. The apparatus according to any one of Illustrative Examples 10-14, wherein the spatial location of one or more pixels of the 2D slice includes the coordinates of one or more pixels.

[0110] Illustrative Example 15. A non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform operations including: receiving one or more 3D (three-dimensional) medical images, each including a plurality of 2D (two-dimensional) slices; extracting a first feature set from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network; resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of the 2D slice from which the corresponding feature was extracted; encoding the resampled first feature set into a second feature set; performing a medical imaging analysis task based on the second feature set; and outputting the results of the medical imaging analysis task.

[0111] Illustrative Example 16. A non-transitory computer-readable storage medium according to Illustrative Example 15, wherein the one or more 3D medical images have at least one of different resolutions, different acquisition orientations, or different domains.

[0112] Illustrative Example 17. A non-transitory computer-readable storage medium according to any one of Illustrative Examples 15-16, wherein resampling each corresponding feature of a first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: encoding the first feature set using localization embedding.

[0113] Illustrative Example 18. A non-transitory computer-readable storage medium according to any one of Illustrative Examples 15-17, wherein resampling each corresponding feature of a first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: encoding the first feature set using modal embedding.

[0114] Illustrative Example 19. A non-transitory computer-readable storage medium according to any one of Illustrative Examples 15-18, wherein encoding a resampled first feature set into a second feature set comprises: encoding the resampled first feature set into a second feature set using at least one of a transformer encoder or a state-space model.

[0115] Illustrative Example 20. A non-transitory computer-readable storage medium according to any one of Illustrative Examples 15-19, wherein performing a medical imaging analysis task based on a second feature set includes: performing the medical imaging analysis task using at least one of a transformer encoder or a state-space model.

Claims

1. A computer-implemented method, comprising: Receive one or more 3D (three-dimensional) medical images, each comprising multiple 2D (two-dimensional) slices; A first feature set is extracted from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network; Based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted, each corresponding feature of the first feature set is resampled; Encode the resampled first feature set into the second feature set; Perform medical imaging analysis tasks based on the second feature set; as well as Output the results of the medical imaging analysis task.

2. The computer-implemented method according to claim 1, wherein the one or more 3D medical images have at least one of different resolutions, different acquisition orientations, or different domains.

3. The computer-implemented method of claim 1, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: Each corresponding feature of the first feature set is resampled onto a 2D uniform pixel grid to form a 2D image in the feature space.

4. The computer-implemented method of claim 3, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted further comprises: Stack 2D images in feature space to form one or more 3D isotropic images in feature space.

5. The computer-implemented method of claim 1, wherein the spatial location of the one or more pixels of the 2D slice includes the coordinates of the one or more pixels.

6. The computer-implemented method of claim 1, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: The first feature set is encoded using localization embedding.

7. The computer-implemented method of claim 1, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: The first feature set is encoded using modality embedding.

8. The computer-implemented method according to claim 1, wherein encoding the resampled first feature set into a second feature set comprises: The first feature set of the resampled model is encoded into a second feature set using at least one of the transformer encoder or the state-space model.

9. The computer-implemented method according to claim 1, wherein performing the medical imaging analysis task based on the second feature set comprises: Perform medical imaging analysis tasks using at least one of a transformer encoder or a state-space model.

10. An apparatus comprising: A component for receiving one or more 3D (three-dimensional) medical images, each of which includes multiple 2D (two-dimensional) slices; Components for extracting a first feature set from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network; A component for resampling each corresponding feature of a first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted; The component used to encode the resampled first feature set into the second feature set; Components used to perform medical imaging analysis tasks based on a second feature set; as well as A component used to output the results of medical imaging analysis tasks.

11. The apparatus of claim 10, wherein the one or more 3D medical images have at least one of different resolutions, different acquisition orientations, or different domains.

12. The apparatus of claim 10, wherein the component for resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: A component for resampling each corresponding feature of a first feature set onto a 2D uniform pixel grid to form a 2D image in feature space.

13. The apparatus of claim 12, wherein the component for resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted further comprises: A component used to stack 2D images in a feature space to form one or more 3D isotropic images in the feature space.

14. The apparatus of claim 10, wherein the spatial location of the one or more pixels of the 2D slice includes the coordinates of the one or more pixels.

15. A non-transitory computer-readable storage medium comprising instructions, which, when executed by a computer, cause the computer to perform operations including: Receive one or more 3D (three-dimensional) medical images, each comprising multiple 2D (two-dimensional) slices; A first feature set is extracted from the plurality of 2D slices of the one or more 3D medical images using a machine learning-based encoding network; Based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted, each corresponding feature of the first feature set is resampled; Encode the resampled first feature set into the second feature set; Perform medical imaging analysis tasks based on the second feature set; as well as Output the results of the medical imaging analysis task.

16. The non-transitory computer-readable storage medium of claim 15, wherein the one or more 3D medical images have at least one of different resolutions, different acquisition orientations, or different domains.

17. The non-transitory computer-readable storage medium of claim 15, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: The first feature set is encoded using localization embedding.

18. The non-transitory computer-readable storage medium of claim 15, wherein resampling each corresponding feature of the first feature set based on the spatial location of one or more pixels of a 2D slice from which corresponding features are extracted comprises: The first feature set is encoded using modality embedding.

19. The non-transitory computer-readable storage medium of claim 15, wherein encoding the resampled first feature set into a second feature set comprises: The first feature set of the resampled model is encoded into a second feature set using at least one of the transformer encoder or the state-space model.

20. The non-transitory computer-readable storage medium of claim 15, wherein performing a medical imaging analysis task based on a second feature set comprises: Perform medical imaging analysis tasks using at least one of a transformer encoder or a state-space model.

Images