Pixel-by-pixel position embedding for medical images in visual transducers

Abstract

Systems and methods are provided for performing medical imaging analysis tasks based on pixel-by-pixel position encoding features. One or more input medical images are received. One or more pixel-by-pixel position embedded images are generated for one or more input medical images using a spatial variation function. A tile is extracted from one or more input medical images and one or more pixel-by-pixel position embedded images. A tile extracted from the one or more input medical images and a corresponding tile extracted from the one or more pixel-by-pixel position embedded images are encoded into a pixel-by-pixel position encoded feature. Medical imaging analysis tasks are performed based on the per-pixel position encoding features using a machine learning based network. And outputting the result of the medical imaging analysis task.

Description

Technical Field

[0001] The present invention relates generally to visual transformers for medical imaging analysis, and in particular to pixel-by-pixel position embedding of medical images for use in visual transformers. Background Art

[0002] A visual transformer is a type of neural network used to perform various computer vision tasks on an input image. Position embeddings are provided to the visual transformer to provide information about the relative or absolute positions of patches of the input image. Conventional position embedding techniques are designed for natural images and do not inherently capture absolute scale or position, but rather only embed relative patch-by-patch position information for each patch in the input image. Tomographic medical images inherently capture absolute scale and position, which is used to improve comparisons between images between or within imaging sessions. However, for clinical and physical reasons, the relationship between image pixels and physical coordinates of tomographic medical images varies between images and is therefore not preserved by conventional position embedding techniques. Summary of the invention

[0003] According to one or more embodiments, a system, method, and non-transitory computer-readable medium storing computer program instructions for performing a medical imaging analysis task based on pixel-by-pixel position encoding features are provided. One or more input medical images are received. One or more pixel-by-pixel position embedding images are generated for the one or more input medical images using a spatial variation function. Patches are extracted from the one or more input medical images and the one or more pixel-by-pixel position embedding images. Patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedding images are encoded into pixel-by-pixel position encoding features. A medical imaging analysis task is performed based on the pixel-by-pixel position encoding features using a network based on machine learning. The result of the medical imaging analysis task is output.

[0004] In one embodiment, generating one or more pixel-by-pixel position embedded images includes sampling a spatially varying function at the position of each pixel of the one or more input medical images. The spatially varying function may be a sinusoidal function. The spatially varying function may be in a reference coordinate system defined relative to the one or more input medical images. The reference coordinate system may include a physical coordinate system of an image acquisition device that acquired the one or more input medical images.

[0005] In one embodiment, the one or more input medical images include a plurality of input medical images, and the one or more pixel-by-pixel position embedded images include a plurality of pixel-by-pixel position embedded images. Each patch extracted from the plurality of input medical images is combined with its corresponding patch extracted from the plurality of pixel-by-pixel position embedded images. The combined patches are separately encoded.

[0006] In one embodiment, the one or more input medical images include a plurality of input medical images, and the one or more pixel-by-pixel position embedded images include a single pixel-by-pixel position embedded image. Corresponding patches extracted from each of the plurality of input medical images and from the single pixel-by-pixel position embedded image are combined. The combined patches are separately encoded.

[0007] In one embodiment, the patches extracted from one or more input medical images are further encoded using the patch-by-patch position embedding features to generate patch-by-patch and pixel-by-pixel position encoding features. A medical imaging analysis task is performed based on the patch-by-patch and pixel-by-pixel position encoding features.

[0008] In one embodiment, the machine learning based network is a vision transformer network.

[0009] These and other advantages of the present invention will become apparent to those of ordinary skill in the art by reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A method for performing medical imaging analysis tasks based on pixel-by-pixel position embedding using a machine learning based network in accordance with one or more embodiments is shown; Figure 2 shows a two-dimensional pixel-by-pixel position embedding image generated by sampling a 3D spatial sine function according to one or more embodiments; Figure 3 shows a schematic diagram visually illustrating the calculation of pixel-by-pixel position embedding values ​​according to one or more embodiments; Figure 4 A workflow for performing a medical imaging analysis task by separately encoding each patch extracted from an input medical image from its corresponding patch extracted from a pixel-by-pixel position embedded image is shown in accordance with one or more embodiments; Figure 5 A workflow for performing a medical imaging analysis task by encoding corresponding patches extracted from an input medical image and from a pixel-by-pixel position embedded image is shown in accordance with one or more embodiments; Figure 6 shows a table comparing the performance of the embodiments described herein with conventional methods; Figure 7 An exemplary artificial neural network that can be used to implement one or more embodiments is shown; Figure 8 shows a convolutional neural network that can be used to implement one or more embodiments; and Fig. 9A high-level block diagram of a computer that can be used to implement one or more embodiments is shown. DETAILED DESCRIPTION

[0011] The present invention generally relates to methods and systems for utilizing pixel-by-pixel position embedding of medical images in a visual transformer. Embodiments of the present invention are described herein to provide a visual understanding of such methods and systems. A digital image is typically composed of digital representations of one or more objects (or shapes). The digital representations of objects are typically described herein in terms of identifying and manipulating the objects. Such manipulations are virtual manipulations performed in the memory or other circuits / hardware of a computer system. Therefore, it is to be understood that embodiments of the present invention may be performed within a computer system using data stored within the computer system. In addition, image pixels referred to herein may equally refer to voxels of an image, and vice versa.

[0012] Embodiments described herein provide a visual transformer network for performing a medical imaging analysis task based on a pixel-by-pixel position embedding of one or more input medical images. The pixel-by-pixel position embedding represents position information (e.g., origin, pixel spacing, direction, physical extent, size, etc.) that is determined for each voxel of one or more input medical images using a spatial variation function. The spatial variation function is defined in a reference coordinate system relative to the one or more input medical images. The pixel-by-pixel position embedding is represented in a pixel-by-pixel position embedding image that is encoded into a pixel-by-pixel position encoded feature together with the one or more input medical images for use by a visual transformer network to perform a medical imaging analysis task.

[0013] Advantageously, the embodiments described herein utilize pixel-by-pixel position embeddings, rather than the patch-by-patch position embeddings utilized in conventional methods. Pixel-by-pixel position embeddings encode more detailed information about the position, scale, orientation, pixel spacing, field of view, etc. of one or more input medical images than conventional patch-by-patch position information. Utilizing pixel-by-pixel position embeddings, the visual transformer network can more efficiently extract information and learn corresponding positional relationships between pairs of input medical images, thereby producing a better performing visual transformer network. Furthermore, utilizing pixel-by-pixel position embeddings, the visual transformer network will have the flexibility to obtain input medical images acquired using different geometries without the need for resampling and resulting information that may degrade network performance.

[0014] Figure 1 A method 100 for performing a medical imaging analysis task based on pixel-by-pixel position embedding using a machine learning-based network is shown in accordance with one or more embodiments. The steps of the method 100 may be performed by one or more suitable computing devices (such as, for example, Fig. 9 Computer 902) to execute.

[0015] exist Figure 1 At step 102, one or more input medical images are received. The one or more input medical images may depict anatomical objects, such as, for example, organs, blood vessels, bones, tumors or other abnormalities, or any other anatomical objects of interest of the patient. The one or more input medical images may be any suitable modality, such as, for example, CT (computed tomography), MRI (magnetic resonance imaging), US (ultrasound), x-ray, or any other medical imaging modality or combination of medical imaging modalities. The one or more input medical images may be 2D (two-dimensional) images and / or 3D (three-dimensional) volumes.

[0016] The one or more input medical images include metadata of position information of each pixel, which is recorded by the image acquisition device during the acquisition of the one or more input medical images. The position information is defined with respect to a physical coordinate system of the image acquisition device. The position information may include, for example, an origin (e.g., coordinates of the first pixel of the image in the physical coordinate system), pixel spacing (e.g., pixel size in each dimension of the image matrix), direction (e.g., an affine matrix mapping image matrix indexes to coordinates in the physical coordinate system), physical limits (e.g., field of view of the image), size (e.g., image matrix size), etc. The one or more input medical images are in a medical image format such as, for example, DICOM (Digital Imaging and Communications in Medicine), MHD (MetaImage MetaHeader), NIFTI (Neuroimaging Informatics Technology Initiative), etc., and the metadata may be stored in a header of the one or more input medical images according to the medical image format. Based on the origin and pixel spacing, the center of each pixel in the physical coordinate system is derived.

[0017] One or more input medical images may be directly obtained from an image acquisition device (e.g., Fig. 9 The image can be received by an image acquisition device 914), such as a CT or MRI scanner, by storing the image in a storage device or memory ( Fig. 9 A previously acquired medical image may be received by loading the memory 910 of the computer 902 or may be received from a remote computer system (e.g., Fig. 9 Computer 902) is received.

[0018] exist Figure 1 At step 104, one or more pixel-by-pixel position embedding images are generated for one or more input medical images using the spatial variation function. In one embodiment, the one or more pixel-by-pixel position embedding images are generated by sampling the spatial variation function at the position of each pixel of the one or more input medical images.

[0019] The spatial variation function is defined in a reference coordinate system relative to one or more input medical images. In one embodiment, the reference coordinate system is the physical coordinate system of the image acquisition device that acquired the one or more input medical images. In another embodiment, the reference coordinate system is defined relative to the one or more input medical images by aligning a canonical reference system to the content of the one or more input medical images by, for example, image-based or landmark-based image registration. In this embodiment, the pixel-by-pixel position embedding values ​​of pixels depicting a particular anatomical object (e.g., the left retina) will be consistent between imaging sessions, even if the patient is positioned differently. The reference coordinate system can be any other suitable coordinate system.

[0020] In one embodiment, the spatially varying function comprises a 3D spatial sine function defined in a reference coordinate system and centered at the origin of the reference coordinate system. The value of the spatially varying function at each position is a function of the coordinates of the position in the reference coordinate system. An example of a 3D spatial sine function is equation (1): Figure 2 A two-dimensional pixel-by-pixel position embedding image 200 generated by sampling the 3D spatial sine function of equation (1) according to one or more embodiments is shown. Pixel-by-pixel position embedding image 200 depicts the sine function with pixel-by-pixel position embedding values ​​calculated according to equation (1).

[0021] Another example of a sine function in 3D space is equation (2): In equations (1) and (2), the hyperparameter A represents the amplitude, ω represents the angular frequency, represents the phase, and x, y and z are the coordinates of the pixel. The spatially varying function may include any other suitable sinusoidal or non-sinusoidal function.

[0022] The spatially varying function is sampled at the coordinates of the center of each corresponding pixel of one or more input medical images in a reference coordinate system to determine a pixel-by-pixel position embedding value. For each input medical image, after determining the pixel-by-pixel position embedding values ​​at all pixel center positions, the pixel-by-pixel position embedding values ​​form an embedded image having the same pixel grid as the input medical image. Figure 3A schematic diagram 300 visually illustrating the calculation of pixel-by-pixel position embedding values ​​according to one or more embodiments is shown. Schematic diagram 300 is schematically shown for one dimension (i.e., the x dimension). Waveform 302 is a 3D spatial sine function in the x dimension, which depicts the relationship between the pixel-by-pixel position embedding value (on the y axis) and the physical position of the pixel along the x axis (on the x axis). Image 1 304 includes a row with six pixels. Image 2 306 shows a row in which the six pixels of image 1 304 are offset by half a pixel size. Image 3 308 shows a high-resolution version of image 1 304, including 12 pixels of half a pixel size. For each corresponding pixel of images 304, 306 and 308, a pixel-by-pixel position embedding value is sampled from waveform 302 as a value on the y axis that intersects with the position of the corresponding pixel along the x axis in waveform 302. Each pixel of image 1 304 is extracted at the "X" symbol on the waveform 302, each pixel of image 2 306 is extracted at the "O" symbol on the waveform 302, and each pixel of image 3 308 is extracted at the "+" symbol on the waveform 302. Image 1 304, image 2 306, and image 3 308 are acquired at different origins or resolutions, but the patient is in the same position. In this way, the value embedded in the pixel-by-pixel position reflects the acquisition difference of images 302, 304, 306 and the relative position relationship of their pixels.

[0023] exist Figure 1 At step 106, patches are extracted from the one or more input medical images and the one or more pixel-by-pixel position embedded images. The patches can be extracted by, for example, cropping the one or more input medical images and the one or more pixel-by-pixel position embedded images into a series of non-overlapping patches with a predetermined fixed size.

[0024] exist Figure 1 At step 108, patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images are encoded into pixel-by-pixel position encoded features. Patches extracted from the one or more input medical images and patches extracted from the one or more pixel-by-pixel position embedded images correspond when they are associated with the same position in a reference coordinate system. Patches can be encoded using any suitable encoder. In one example, the encoder is an encoder network based on machine learning (e.g., CNN, convolutional neural network). In another example, the encoder is a linear projection encoder.

[0025] In one embodiment, for example, where the one or more input medical images is a single input medical image and the one or more pixel-by-pixel position embedded images is a single pixel-by-pixel position embedded image, each patch extracted from the single input medical image is combined (e.g., added or concatenated in the channel dimension) with its corresponding patch extracted from the single pixel-by-pixel position embedded image, and each combined patch is separately encoded by the encoder.

[0026] In one embodiment, for example, where the one or more input medical images are multiple input medical images (e.g., in different imaging modalities or from multiple MR sequences), if the input medical images have different spatial relationships in different spatial coordinate systems, each patch extracted from the multiple input medical images is combined with its corresponding patch extracted from the pixel-by-pixel position embedded image, and the combined patches are separately encoded by the encoder. Figure 4 A workflow 400 for performing a medical imaging analysis task by separately encoding each patch extracted from an input medical image from its corresponding patch extracted from a pixel-by-pixel position embedded image is shown in accordance with one or more embodiments. Patch 402 extracted from a contrast 1 image and patch 406 extracted from a contrast 2 image have different spatial relationships. Patches 402 and 406 are combined with their corresponding patches 404 and 408 extracted from the pixel-by-pixel position embedded image, respectively, and the combined patches are encoded by encoder 408 to generate pixel-by-pixel position encoded features 410 and 412. The following will describe in more detail Figure 4 .

[0027] In one embodiment, for example, where the one or more input medical images are multiple input medical images, if the input medical images have the same spatial relationship in the same spatial coordinate system, the pixel-by-pixel position embedded images are the same, and thus there is a single pixel-by-pixel position embedded image. The corresponding patches extracted from each of the multiple input medical images and from the pixel-by-pixel position embedded image are combined, and each combined patch is separately encoded by the encoder. Figure 5 A workflow 500 for performing a medical imaging analysis task by encoding corresponding patches extracted from an input medical image and from a single pixel-by-pixel position embedded image according to one or more embodiments is shown. Patches 506 extracted from the contrast 1 image and patches 504 extracted from the contrast 2 image have the same spatial relationship. Corresponding patches 506 and 504 and patches 502 extracted from the pixel-by-pixel position embedded image are combined, and the combined patches are separately encoded by an encoder 508 to generate pixel-by-pixel position encoded features 510. The following will describe in more detail Figure 5 .

[0028] exist Figure 1At step 110, a medical imaging analysis task is performed based on pixel-by-pixel position encoding features using a machine learning-based network. The medical imaging analysis task may be any medical imaging analysis task, such as, for example, detection, classification, segmentation, and the like.

[0029] In one embodiment, the machine learning based network is a visual transformer network. However, the machine learning based network can be any suitable machine learning based network. The machine learning based network receives pixel-by-pixel position encoded features as input and generates results for a medical imaging task as output. The machine learning based network utilizes the position information encoded in the pixel-by-pixel position encoded features to perform the medical imaging analysis task. The machine learning based network is trained during a previous offline or training phase. Once trained, the machine learning based network is applied during an online or inference phase, for example to perform Figure 1 Step 110. In one example, the machine learning based network is Figure 4 The converter encoder 414 or Figure 5 Converter encoder 512.

[0030] exist Figure 1 At step 112, the result of the medical imaging analysis task is output. For example, the result of the medical imaging analysis task may be output by displaying the result of the medical imaging analysis task on a display device of the computer system, storing the result of the medical imaging analysis task on a memory or storage device of the computer system, or transmitting the result of the medical imaging analysis task to a remote computer system.

[0031] In one embodiment, Figure 1 At step 108, the patches extracted from the one or more input medical images are additionally encoded using the patch-by-patch position embedding features to generate patch-by-patch and pixel-by-pixel position encoding features. For example, the patch-by-patch position embedding features may be generated, for example, according to known techniques. Figure 1 At step 110, a medical imaging analysis task is performed based on the patch-by-patch and pixel-by-pixel position encoded features.

[0032] Advantageously, pixel-by-pixel position embedding according to the embodiments described herein provides input flexibility. Using pixel-by-pixel position embedding, the visual transformer has the ability to receive input images obtained using different spatial relationships between voxels and a spatial coordinate system without resampling and without losing the spatial relationship. In this way, resampling that may cause information loss and impair network performance can be avoided. In addition, pixel-by-pixel position embedding provides pixel-level position relationships. The pixel-by-pixel position embedding value depends on the position of each pixel in a reference coordinate system. The difference in the pixel-by-pixel position embedding of two input images can reflect the difference in their pixel grids. Therefore, the pixel-by-pixel position embedding can provide the visual transformer with the relative position information of the pixel grid of each patch in 3D space. In this way, a visual transformer trained using pixel-by-pixel position embedding can potentially learn the positional relationship of each patch down to the pixel level to accurately extract and align information from each patch and achieve better performance.

[0033] The embodiments described herein are experimentally validated on the medical imaging analysis task of acute / subacute cerebral infarction segmentation using a dataset of 1142, 133, and 143 subjects in the training, validation, and test splits, demonstrating the performance improvement due to pixel-wise position embedding. Figure 6 A table 600 comparing the performance of the embodiments described herein with conventional methods is shown. Table 600 shows that the proposed position embedding (representing pixel-by-pixel position embedding according to the embodiments described herein) outperforms both UNETR (UNet Transformer) and the original position embedding of Swin UNETR (representing conventional patch-by-patch position embedding).

[0034] The embodiments described herein are described with respect to the claimed system and the claimed method. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, the features described or claimed in the context of the method may be utilized to improve the claims for the system. In this case, the functional features of the method are embodied by providing the target unit of the system.

[0035] In addition, certain embodiments described herein are described with respect to methods and systems for utilizing trained machine learning-based models, and with respect to methods and systems for training machine learning-based models. Features, advantages, or alternative embodiments herein may be assigned to other claimed objects, and vice versa. In other words, features described or claimed in the context of methods and systems for utilizing trained machine learning-based models may be utilized to improve claims for methods and systems for training machine learning-based models, and vice versa.

[0036] In particular, the trained machine learning-based models applied in the embodiments described herein can be adapted by methods and systems for training machine learning-based models. In addition, the input data of the trained machine learning-based models can include advantageous features and embodiments of the training input data, and vice versa. In addition, the output data of the trained machine learning-based models can include advantageous features and embodiments of the output training data, and vice versa.

[0037] Generally speaking, trained machine learning-based models mimic the cognitive functions of humans in connection with other human minds. In particular, through training based on training data, trained machine learning-based models are able to adapt to new situations and detect and extrapolate patterns.

[0038] In general, the parameters of the machine learning based model can be adapted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. In addition, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of the trained machine learning based model can be iteratively adapted through several training steps.

[0039] In particular, the trained machine learning based model may include a neural network, a support vector machine, a decision tree and / or a Bayesian network, and / or the trained machine learning based model may be based on k-means clustering, Q learning, a genetic algorithm and / or an association rule. In particular, the neural network may be a deep neural network, a convolutional neural network or a convolutional deep neural network. In addition, the neural network may be an adversarial network, a deep adversarial network and / or a generative adversarial network.

[0040] Figure 7 An embodiment of an artificial neural network 700 according to one or more embodiments is shown. Alternative terms for "artificial neural network" are "neural network", "artificial neural network" or "neural network". The machine learning networks described herein, such as those that can be used in Figure 1 The machine learning based encoder network used at step 108, or the machine learning based network used at step 110, Figure 4 CNN encoder 408 or Figure 5 The CNN encoder 508 can be implemented using an artificial neural network 700.

[0041] The artificial neural network 700 includes nodes 702-722 and edges 732, 734, ... 736, wherein each edge 732, 734, ... 736 is a directed connection from a first node 702-722 to a second node 702-722. Generally speaking, the first node 702-722 and the second node 702-722 are different nodes 702-722, or the first node 702-722 and the second node 702-722 are the same. For example, in Figure 7 , edge 732 is a directed connection from node 702 to node 706, and edge 734 is a directed connection from node 704 to node 706. Edges 732, 734, ... 736 from a first node 702-722 to a second node 702-722 are also denoted as "incoming edges" to the second node 702-722, and as "outgoing edges" to the first node 702-722.

[0042] In this embodiment, the nodes 702-722 of the artificial neural network 700 may be arranged in layers 724-730, wherein the layers may include an inherent order introduced by edges 732, 734 ... 736 between the nodes 702-722. In particular, the edges 732, 734 ... 736 may only exist between adjacent layers of nodes. Figure 7 In the embodiment shown in , there is an input layer 724 that includes only nodes 702 and 704 without incoming edges, an output layer 730 that includes only nodes 722 without outgoing edges, and hidden layers 726, 728 between the input layer 724 and the output layer 730. In general, the number of hidden layers 726, 728 can be selected arbitrarily. The number of nodes 702 and 704 within the input layer 724 is generally related to the number of input values ​​of the neural network 700, and the number of nodes 722 within the output layer 730 is generally related to the number of output values ​​of the neural network 700.

[0043] In particular, a (real) number may be assigned as a value to each node 702-722 of the neural network 700. Here, x (n) i represents the value of the i-th node 702-722 of the n-th layer 724-730. The value of the node 702-722 of the input layer 724 is equivalent to the input value of the neural network 700, and the value of the node 722 of the output layer 730 is equivalent to the output value of the neural network 700. In addition, each edge 732, 734...736 may include a weight as a real number, in particular, the weight is a real number in the interval [-1,1] or in the interval [0,1]. Here, w (m,n) i,jrepresents the weight of the edge between the i-th node 702-722 of the m-th layer 724-730 and the j-th node 702-722 of the n-th layer 724-730. (n) i,j is defined for the weight w (n,n+1) i,j .

[0044] In particular, in order to calculate the output value of the neural network 700, the input value is propagated through the neural network. In particular, the values ​​of the nodes 702-722 of the (n+1)th layer 724-730 can be calculated based on the values ​​of the nodes 702-722 of the nth layer 724-730 by the following formula:

[0045] In this article, function f is a transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic function, generalized logistic function, hyperbolic tangent function, inverse tangent function, error function, smooth step function), or rectifier functions. Transfer functions are mainly used for normalization purposes.

[0046] In particular, these values ​​are propagated through the neural network layer by layer, where the value of the input layer 724 is given by the input of the neural network 700, where the value of the first hidden layer 726 can be calculated based on the value of the input layer 724 of the neural network, where the value of the second hidden layer 728 can be calculated based on the value of the first hidden layer 726, and so on.

[0047] To set the value of edge w (m,n) i,j , training data must be used to train the neural network 700. In particular, the training data includes training input data and training output data (denoted as t i ). For the training step, the neural network 700 is applied to the training input data to generate the calculated output data. In particular, the training data and the calculated output data include a certain number of values, which is equal to the number of nodes of the output layer.

[0048] In particular, the comparison between the calculated output data and the training data is used to recursively adapt the weights within the neural network 700 (back propagation algorithm). In particular, the weights are changed according to the following formula: where γ is the learning rate and if the (n+1)th layer is not the output layer, it can be based on δ (n+1) j Recursively add the number δ (n)j Calculated as: And if the (n+1)th layer is the output layer 730, it is calculated as: where f' is the first derivative of the activation function, and y (n+1) j is the comparison training value of the j-th node of the output layer 730.

[0049] Figure 8 A convolutional neural network 800 is shown according to one or more embodiments. The machine learning networks described herein, such as those that can be used in Figure 1 The machine learning based encoder network used at step 108, or the machine learning based network used at step 110, Figure 4 CNN encoder 408 or Figure 5 The CNN encoder 508 can be implemented using a convolutional neural network 800.

[0050] exist Figure 8 In the embodiment shown in , the convolutional neural network 800 includes an input layer 802, a convolutional layer 804, a pooling layer 806, a fully connected layer 808, and an output layer 810. Alternatively, the convolutional neural network 800 may include several convolutional layers 804, several pooling layers 806, and several fully connected layers 808, as well as other types of layers. The order of these layers can be arbitrarily selected, and the fully connected layer 808 is usually used as the last layer before the output layer 810.

[0051] In particular, within the convolutional neural network 800, the nodes 812-820 of a layer 802-810 may be considered to be arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of the nodes 812-820 indexed by i and j in the nth layer 802-810 may be represented as x (n) [i,j] However, the arrangement of the nodes 812-820 of a layer 802-810 has no effect on the computations performed within the convolutional neural network 800 itself, since they are given only by the structure and weights of the edges.

[0052] In particular, the convolution layer 804 is characterized by the structure and weights of the incoming edges that form the convolution operation based on a certain number of kernels. In particular, the structure and weights of the incoming edges are selected so that the value x of the node 814 of the convolution layer 804 is (n) k Based on the value x of node 812 of the previous layer 802 (n-1) is calculated as the convolution x (n) k =Kk *x (n-1) , where the convolution* in two dimensions is defined as:

[0053] Here, the kth kernel K k Is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 812-818 (e.g., a 3×3 matrix or a 5×5 matrix). In particular, this means that the weights of the incoming edges are not independent, but are selected so that they produce the convolution formula. In particular, for the kernel as a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to an independent weight), regardless of the number of nodes 812-820 in the corresponding layer 802-810. In particular, for a convolutional layer 804, the number of nodes 814 in the convolutional layer is equal to the number of nodes 812 in the previous layer 802 multiplied by the number of kernels.

[0054] If the nodes 812 of the previous layer 802 are arranged as a d-dimensional matrix, the use of multiple kernels can be interpreted as adding an additional dimension (denoted as the "depth" dimension), so that the nodes 814 of the convolution layer 814 are arranged as a (d+1)-dimensional matrix. If the nodes 812 of the previous layer 802 are already arranged as a (d+1)-dimensional matrix including the depth dimension, the use of multiple kernels can be interpreted as expanding along the depth dimension, so that the nodes 814 of the convolution layer 804 are also arranged as a (d+1)-dimensional matrix, where the size of the (d+1)-dimensional matrix with respect to the depth dimension is the number of kernels times the size in the previous layer 802.

[0055] An advantage of using convolutional layers 804 is that spatially local correlations in the input data can be exploited by enforcing local connectivity patterns between nodes in adjacent layers, in particular by connecting each node only to a small region of nodes in the previous layer.

[0056] exist Figure 8 In the embodiment shown in , the input layer 802 includes 36 nodes 812 arranged as a two-dimensional 6×6 matrix. The convolution layer 804 includes 72 nodes 814 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with the kernel. Equivalently, the nodes 814 of the convolution layer 804 can be interpreted as being arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0057] The pooling layer 806 is characterized by the activation function of its nodes 816 and the structure and weights of the incoming edges that form the pooling operation based on the nonlinear pooling function f. For example, in the two-dimensional case, the value x of the node 816 of the pooling layer 806 (n) It can be based on the value x of the node 814 of the previous layer 804 (n-1) and is calculated as: x (n) [i, j] = f(x (n-1) [id1, jd2], ..., x (n-1) [id1+d1-1, jd2+d2-1]).

[0058] In other words, by using the pooling layer 806, the number of nodes 814, 816 can be reduced by replacing the number d1·d2 of neighboring nodes 814 in the previous layer 804 with a single node 816 in the pooling layer, which is calculated based on the values ​​of the number of neighboring nodes. In particular, the pooling function f can be a maximum function, an average value, or an L2-norm. In particular, for the pooling layer 806, the weights of the incoming edges are fixed and are not modified by training.

[0059] The advantage of using the pooling layer 806 is that the number of nodes 814, 816 and the number of parameters are reduced. This results in a reduction in the amount of computation in the network and in control of overfitting.

[0060] exist Figure 8 In the embodiment shown in , pooling layer 806 is maximum pooling, replacing four adjacent nodes with only one node whose value is the maximum of the values ​​of the four adjacent nodes. Maximum pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, maximum pooling is applied to each of the two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0061] The fully connected layer 808 is characterized by the fact that most of the edges, in particular all of the edges, between the nodes 816 of the previous layer 806 and the nodes 818 of the fully connected layer 808 exist, and the weight of each of the edges can be adjusted individually.

[0062] In this embodiment, the nodes 816 of the previous layer 806 of the fully connected layer 808 are displayed both as a two-dimensional matrix and additionally as unrelated nodes (which are indicated as a row of nodes with a reduced number of nodes for better presentability). In this embodiment, the number of nodes 818 in the fully connected layer 808 is equal to the number of nodes 816 in the previous layer 806. Alternatively, the number of nodes 816, 818 may be different.

[0063] Furthermore, in this embodiment, the values ​​of the nodes 820 of the output layer 810 are determined by applying the Softmax function to the values ​​of the nodes 818 of the previous layer 808. By applying the Softmax function, the sum of the values ​​of all the nodes 820 of the output layer 810 is 1, and all the values ​​of all the nodes 820 of the output layer are real numbers between 0 and 1.

[0064] The convolutional neural network 800 may also include a ReLU (rectified linear unit) layer or an activation layer having a nonlinear transfer function. In particular, the number of nodes and the structure of the nodes included in the ReLU layer are equivalent to the number of nodes and the structure of the nodes included in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a rectification function to the value of the corresponding node of the previous layer.

[0065] The inputs and outputs of different CNN blocks can be wired using summation (residual / dense NNs), element-wise multiplication (attention), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the CNN architecture can be nested instead of sequential.

[0066] In particular, the convolutional neural network 800 can be trained based on a back-propagation algorithm. To prevent overfitting, regularization methods such as dropout of nodes 812-820, random pooling, use of artificial data, weight decay based on L1 or L2 norms, or maximum norm constraints can be used. Different loss functions can be combined for training the same neural network to reflect joint training objectives. A subset of neural network parameters can be excluded from optimization to retain weights pre-trained on another data set.

[0067] The systems, devices, and methods described herein may be implemented using digital circuits, or may be implemented using one or more computers using known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include or may be coupled to one or more mass storage devices, such as one or more disks, internal hard disks and removable disks, magneto-optical disks, optical disks, and the like.

[0068] The systems, devices, and methods described herein can be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computer is located remotely from the server computer and interacts via a network. The client-server relationship can be defined and controlled by computer programs running on the respective client and server computers.

[0069] The systems, devices, and methods described herein can be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor connected to the network communicates with one or more client computers via the network. The client computer can communicate with the server via, for example, a web browser application that resides and operates on the client computer. The client computer can store data on the server and access the data via the network. The client computer can transmit a request for data or a request for an online service to the server via the network. The server can perform the requested service and provide the data to (one or more) client computers. The server can also transmit data adapted to cause the client computer to perform a specified function (e.g., perform a calculation, display specified data on a screen, etc.). For example, the server can transmit data adapted to cause the client computer to perform one or more steps or functions (including Figure 1 or one or more steps or functions of 3-5). Certain steps or functions of the methods and workflows described herein (including Figure 1 or one or more steps or functions of 3-5) can be performed by a server or by another processor in a network-based cloud computing system. Certain steps or functions of the methods and workflows described herein (including Figure 1 or one or more steps 3-5) can be performed by a client computer in a network-based cloud computing system. The steps or functions of the methods and workflows described herein (including Figure 1 Or one or more steps 3-5) can be performed by a server and / or by a client computer in a network-based cloud computing system in any combination.

[0070] The systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier (e.g., embodied in a non-transitory machine-readable storage device) for execution by a programmable processor; and the methods and workflow steps described herein (including Figure 1 or one or more steps or functions of 3-5) can be implemented using one or more computer programs executable by such a processor. A computer program is a set of computer program instructions that can be used directly or indirectly in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language (including compiled or interpreted languages), and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0071] Fig. 9A high-level block diagram of an example computer 902 that can be used to implement the systems, apparatus, and methods described herein is depicted in FIG. The computer 902 includes a processor 904 operatively coupled to a data storage device 912 and a memory 910. The processor 904 controls the overall operation of the computer 902 by executing computer program instructions that define such operation. The computer program instructions may be stored in the data storage device 912, or other computer-readable medium, and loaded into the memory 910 when execution of the computer program instructions is desired. Thus, Figure 1 The method and workflow steps or functions of 3-5 may be defined by computer program instructions stored in the memory 910 and / or the data storage device 912, and may be controlled by the processor 904 executing the computer program instructions. For example, the computer program instructions may be implemented as a program programmed by a person skilled in the art to execute Figure 1 or 3-5 method and workflow steps or functions of computer executable code. Thus, by executing the computer program instructions, the processor 904 performs Figure 1 The computer 902 may also include one or more network interfaces 906 for communicating with other devices via a network. The computer 902 may also include one or more input / output devices 908 (e.g., a display, keyboard, mouse, speaker, button, etc.) that enable a user to interact with the computer 902.

[0072] The processor 904 may include both general-purpose and special-purpose microprocessors, and may be the sole processor or one of multiple processors of the computer 902. For example, the processor 904 may include one or more central processing units (CPUs). The processor 904, the data storage device 912, and / or the memory 910 may include, be supplemented by, or be incorporated in one or more application-specific integrated circuits (ASICs) and / or one or more field-programmable gate arrays (FPGAs).

[0073] The data storage device 912 and the memory 910 each include a tangible, non-transitory computer-readable storage medium. The data storage device 912 and the memory 910 may each include a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices (such as an internal hard disk and a removable magnetic disk), a magneto-optical disk storage device, an optical disk storage device, a flash memory device, a semiconductor memory device (such as an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM), a digital versatile disk read-only memory (DVD-ROM) disk) or other non-volatile solid-state storage devices.

[0074] The input / output devices 908 may include peripheral devices such as printers, scanners, display screens, etc. For example, the input / output devices 908 may include a display device (such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user, a keyboard, and a pointing device such as a mouse or trackball through which a user can provide input to the computer 902.

[0075] The image acquisition device 914 can be connected to the computer 902 to input image data (e.g., medical images) to the computer 902. It is possible to implement the image acquisition device 914 and the computer 902 as one device. It is also possible that the image acquisition device 914 and the computer 902 communicate wirelessly via a network. In a possible embodiment, the computer 902 can be remotely located relative to the image acquisition device 914.

[0076] One or more computers, such as computer 902, may be used to implement any or all of the systems and apparatus discussed herein.

[0077] Those skilled in the art will recognize that actual computer or computer system implementations may have other structures and may include other components, and for illustrative purposes, Fig. 9 is a high-level representation of some of the components in such a computer.

[0078] Independent of the use of the grammatical term, individuals with masculine or feminine identities are also included in the term.

[0079] The foregoing specific embodiments should be understood in every respect to be illustrative and exemplary, rather than restrictive, and the scope of the invention disclosed herein is not determined by the specific embodiments, but by the claims interpreted according to the full breadth allowed by the patent law. It is to be understood that the embodiments shown and described herein are merely illustrative of the principles of the invention, and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the invention. Various other feature combinations may be implemented by those skilled in the art without departing from the scope and spirit of the invention.

[0080] The following is a list of non-limiting illustrative embodiments disclosed herein: Illustrative embodiment 1. A computer-implemented method, comprising: receiving one or more input medical images; generating one or more pixel-by-pixel position embedding images for the one or more input medical images using a spatially varying function; extracting patches from the one or more input medical images and the one or more pixel-by-pixel position embedding images; encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedding images into pixel-by-pixel position encoded features; performing a medical imaging analysis task based on the pixel-by-pixel position encoded features using a machine learning-based network; and outputting a result of the medical imaging analysis task.

[0081] Illustrative embodiment 2. The computer-implemented method of illustrative embodiment 1, wherein generating one or more pixel-by-pixel position embedded images for the one or more input medical images using a spatially varying function comprises: sampling the spatially varying function at the position of each pixel of the one or more input medical images.

[0082] Illustrative Embodiment 3. The computer implemented method according to any one of the preceding embodiments, wherein the spatially varying function is a sinusoidal function.

[0083] Illustrative Embodiment 4. The computer-implemented method according to any of the preceding embodiments, wherein the spatially varying function is in a reference coordinate system defined relative to the one or more input medical images.

[0084] Illustrative Embodiment 5. The computer-implemented method according to any of the preceding embodiments, wherein the reference coordinate system comprises a physical coordinate system of an image acquisition device which acquired the one or more input medical images.

[0085] Illustrative embodiment 6. A computer-implemented method according to one of the preceding embodiments, wherein the one or more input medical images comprises a plurality of input medical images, and the one or more pixel-by-pixel position embedded images comprises a plurality of pixel-by-pixel position embedded images, and encoding patches extracted from the one or more input medical images with corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: combining each patch extracted from the plurality of input medical images with its corresponding patch extracted from the plurality of pixel-by-pixel position embedded images; and separately encoding the combined patches.

[0086] Illustrative embodiment 7. A computer-implemented method according to one of the preceding embodiments, wherein the one or more input medical images comprises a plurality of input medical images, and the one or more pixel-by-pixel position embedded images comprises a single pixel-by-pixel position embedded image, and encoding patches extracted from the one or more input medical images with corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: combining the corresponding patches extracted from each of the plurality of input medical images and from the single pixel-by-pixel position embedded image; and separately encoding the combined patches.

[0087] Illustrative embodiment 8. A computer-implemented method according to one of the preceding embodiments, wherein: encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: encoding patches extracted from the one or more input medical images using the patch-by-patch position embedded features to generate patch-by-patch and pixel-by-pixel position encoded features; and performing a medical imaging analysis task based on the pixel-by-pixel position encoded features using a machine learning-based network comprises: performing a medical imaging analysis task based on the patch-by-patch and pixel-by-pixel position encoded features.

[0088] Illustrative Embodiment 9. The computer-implemented method according to any of the preceding embodiments, wherein the machine learning based network is a vision transformer network.

[0089] Illustrative embodiment 10. An apparatus comprising: means for receiving one or more input medical images; means for generating one or more pixel-by-pixel position embedding images for the one or more input medical images using a spatially varying function; means for extracting patches from the one or more input medical images and the one or more pixel-by-pixel position embedding images; means for encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedding images into pixel-by-pixel position encoded features; means for performing a medical imaging analysis task based on the pixel-by-pixel position encoded features using a machine learning-based network; and means for outputting a result of the medical imaging analysis task.

[0090] Illustrative embodiment 11. The apparatus according to illustrative embodiment 10, wherein the means for generating one or more pixel-by-pixel position embedded images for the one or more input medical images using a spatial variation function comprises: means for sampling the spatial variation function at the position of each pixel of the one or more input medical images.

[0091] Illustrative Embodiment 12. The apparatus of any of Illustrative Embodiments 10-11, wherein the spatially varying function is a sinusoidal function.

[0092] Illustrative Embodiment 13. The apparatus of any of Illustrative Embodiments 10-12, wherein the spatially varying function is in a reference coordinate system defined relative to the one or more input medical images.

[0093] Illustrative embodiment 14. The device of any of illustrative embodiments 10-13, wherein the reference coordinate system comprises a physical coordinate system of an image acquisition device that acquired the one or more input medical images.

[0094] Illustrative embodiment 15. A non-transitory computer-readable medium storing computer program instructions, which when executed by a processor causes the processor to perform operations, including: receiving one or more input medical images; generating one or more pixel-by-pixel position embedding images for the one or more input medical images using a spatially varying function; extracting patches from the one or more input medical images and the one or more pixel-by-pixel position embedding images; encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedding images into pixel-by-pixel position encoded features; performing a medical imaging analysis task based on the pixel-by-pixel position encoded features using a machine learning-based network; and outputting a result of the medical imaging analysis task.

[0095] Illustrative embodiment 16. The non-transitory computer readable medium of illustrative embodiment 15, wherein generating one or more pixel-by-pixel position embedded images for the one or more input medical images using a spatially varying function comprises: sampling the spatially varying function at the position of each pixel of the one or more input medical images.

[0096] Illustrative embodiment 17. A non-transitory computer-readable medium according to any one of illustrative embodiments 15-16, wherein the one or more input medical images include multiple input medical images, and the one or more pixel-by-pixel position embedded images include multiple pixel-by-pixel position embedded images, and encoding patches extracted from the one or more input medical images with corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features includes: combining each patch extracted from the multiple input medical images with its corresponding patch extracted from the multiple pixel-by-pixel position embedded images; and separately encoding the combined patches.

[0097] Illustrative embodiment 18. A non-transitory computer-readable medium according to any one of illustrative embodiments 15-17, wherein the one or more input medical images include a plurality of input medical images, and the one or more pixel-by-pixel position embedded images include a single pixel-by-pixel position embedded image, and encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features includes: combining corresponding patches extracted from each of the plurality of input medical images and from the single pixel-by-pixel position embedded image; and separately encoding the combined patches.

[0098] Illustrative embodiment 19. A non-transitory computer-readable medium according to any one of illustrative embodiments 15-18, wherein: encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: encoding patches extracted from the one or more input medical images using the patch-by-patch position embedded features to generate patch-by-patch and pixel-by-pixel position encoded features; and performing a medical imaging analysis task based on the pixel-by-pixel position encoded features using a machine learning-based network comprises: performing a medical imaging analysis task based on the patch-by-patch and pixel-by-pixel position encoded features.

[0099] Illustrative Embodiment 20. The non-transitory computer readable medium of any one of Illustrative Embodiments 15-19, wherein the machine learning based network is a vision transformer network.

Claims

1. A computer-implemented method comprising: receiving one or more input medical images; generating one or more pixel-wise position embedding images for the one or more input medical images using a spatially varying function; extracting patches from the one or more input medical images and the one or more pixel-wise position embedded images; encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-wise position embedded images into pixel-wise position encoded features; Using a machine learning based network to perform a medical imaging analysis task based on the pixel-by-pixel position encoded features; as well as Output the results of the medical imaging analysis task.

2. The computer-implemented method of claim 1 , wherein generating one or more pixel-wise position embedding images for the one or more input medical images using a spatially varying function comprises: The spatially varying function is sampled at the location of each pixel of the one or more input medical images. The computer-implemented method of claim 1 , wherein the spatially varying function is a sinusoidal function. 4 . The computer-implemented method of claim 1 , wherein the spatially varying function is in a reference coordinate system defined relative to the one or more input medical images.

5. The computer-implemented method of claim 4, wherein the reference coordinate system comprises a physical coordinate system of an image acquisition device that acquired the one or more input medical images.

6. The computer-implemented method of claim 1 , wherein the one or more input medical images comprises a plurality of input medical images, and the one or more pixel-by-pixel position embedded images comprises a plurality of pixel-by-pixel position embedded images, and encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: combining each patch extracted from the plurality of input medical images with its corresponding patch extracted from the plurality of pixel-wise position embedded images; as well as The combined tiles are encoded separately.

7. The computer-implemented method of claim 1 , wherein the one or more input medical images comprises a plurality of input medical images and the one or more pixel-by-pixel position embedded images comprises a single pixel-by-pixel position embedded image, and encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: combining corresponding patches extracted from each of the plurality of input medical images and from the single pixel-wise position embedded image; as well as The combined tiles are encoded separately.

8. The computer-implemented method of claim 1 , wherein: Encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: encoding patches extracted from the one or more input medical images using the patch-by-patch position embedded features to generate patch-by-patch and pixel-by-pixel position encoded features; and Using a machine learning based network to perform a medical imaging analysis task based on the pixel-by-pixel position encoding features includes: performing a medical imaging analysis task based on the patch-by-patch and pixel-by-pixel position encoding features.

9. The computer-implemented method of claim 1, wherein the machine learning based network is a vision transformer network.

10. A device comprising: means for receiving one or more input medical images; means for generating one or more pixel-wise position embedding images for the one or more input medical images using a spatially varying function; means for extracting patches from said one or more input medical images and said one or more pixel-wise position embedded images; means for encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-wise position embedded images into pixel-wise position encoded features; means for performing a medical imaging analysis task based on said pixel-by-pixel position encoded features using a machine learning based network; as well as Means for outputting the results of a medical imaging analysis task.

11. The apparatus of claim 10, wherein the means for generating one or more pixel-wise position embedding images for the one or more input medical images using a spatially varying function comprises: Means for sampling a spatially varying function at the location of each pixel of the one or more input medical images.

12. The apparatus of claim 10, wherein the spatially varying function is a sinusoidal function.

13. The apparatus of claim 10, wherein the spatially varying function is in a reference coordinate system defined relative to the one or more input medical images.

14. The apparatus of claim 13, wherein the reference coordinate system comprises a physical coordinate system of an image acquisition device that acquired the one or more input medical images.

15. A non-transitory computer readable medium storing computer program instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving one or more input medical images; generating one or more pixel-wise position embedding images for the one or more input medical images using a spatially varying function; extracting patches from the one or more input medical images and the one or more pixel-wise position embedded images; encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-wise position embedded images into pixel-wise position encoded features; Using a machine learning based network to perform a medical imaging analysis task based on the pixel-by-pixel position encoded features; as well as Output the results of the medical imaging analysis task.

16. The non-transitory computer-readable medium of claim 15, wherein generating one or more pixel-wise position embedding images for the one or more input medical images using a spatially varying function comprises: The spatially varying function is sampled at the location of each pixel of the one or more input medical images.

17. The non-transitory computer-readable medium of claim 15, wherein the one or more input medical images comprises a plurality of input medical images, and the one or more pixel-by-pixel position embedded images comprises a plurality of pixel-by-pixel position embedded images, and encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: combining each patch extracted from the plurality of input medical images with its corresponding patch extracted from the plurality of pixel-wise position embedded images; as well as The combined tiles are encoded separately.

18. The non-transitory computer-readable medium of claim 15, wherein the one or more input medical images comprises a plurality of input medical images, and the one or more pixel-by-pixel position embedded images comprises a single pixel-by-pixel position embedded image, and encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: combining corresponding patches extracted from each of the plurality of input medical images and from the single pixel-wise position embedded image; as well as The combined tiles are encoded separately.

19. The non-transitory computer readable medium of claim 15, wherein: Encoding patches extracted from the one or more input medical images and corresponding patches extracted from the one or more pixel-by-pixel position embedded images into pixel-by-pixel position encoded features comprises: encoding patches extracted from the one or more input medical images using the patch-by-patch position embedded features to generate patch-by-patch and pixel-by-pixel position encoded features; and Using a machine learning based network to perform a medical imaging analysis task based on the pixel-by-pixel position encoding features includes: performing a medical imaging analysis task based on the patch-by-patch and pixel-by-pixel position encoding features.

20. The non-transitory computer readable medium of claim 15, wherein the machine learning based network is a vision transformer network.

Images