Segmenting anatomical structures in medical volume imaging data sets using a transformer

Abstract

This disclosure relates to segmenting anatomical structures in a medical volumetric imaging dataset using a transformer. A framework for segmenting a volumetric imaging dataset is provided. A segmented dataset is obtained by using at least one trained function to locate one or more anatomical features in a patient's medical volumetric imaging dataset. The trained function may include a residual neural network that processes a first feature vector encoding features of the medical volumetric imaging dataset and outputs a second feature vector. The trained function may further include a transformer network that processes a third feature vector obtained by combining the second feature vector with a positional encoding and generates a contextualized fourth feature vector. The trained function may further include a decoder network for generating a segmented dataset based on the fourth feature vector.

Description

[0001] Cross-references to related applications This application claims priority to German application No. 10 2025 114 214.9, filed on April 10, 2025, the contents of which are incorporated herein by reference. Technical Field

[0002] Various examples of this disclosure relate to generating segmentation datasets of one or more anatomical features from a medical volumetric imaging dataset of a localized patient. Specifically, various examples of this disclosure relate to using a machine learning model as a trained function for generating the segmentation dataset. Background Technology

[0003] Multi-organ segmentation of volumetric medical imaging datasets is used in various clinical procedures. However, the time and memory consumption of segmenting the entire three-dimensional volume on a voxel-by-voxel basis using machine learning (ML) models such as deep neural networks (DNNs) can be significant.

[0004] In existing technologies, various DNN architectures have been used to implement multi-organ segmentation. However, voxel-level computations utilizing these DNNs are typically slow. For example, transformer-based methods have reportedly achieved runtimes of up to 60 seconds on graphics processing units (GPUs). Real-time computational efficiency has not yet been achieved, especially on hardware using only a central processing unit (CPU).

[0005] Furthermore, fast coarse segmentation and refinement methods using DNNs based on existing classifiers can be implemented on conventional on-premise personal computer (PC) hardware, including CPUs. However, edge refinement of individual segmentation masks reportedly has an average runtime of 9.51 seconds. Although the system is independent of GPU usage, it is still too slow for near real-time applications, hindering the efficiency of radiologists.

[0006] Various attempts have been made to use sparse sampling and residual DNNs for fast and fine segmentation. However, using residual DNNs alone cannot fully capture the spatial dependencies between descriptors, resulting in unclear edges. Therefore, using only residual DNNs may not be able to fully utilize and analyze spatial dependencies and anatomical features.

[0007] As will be understood from the above, the various techniques reported in the prior art suffer from limited accuracy in segmenting data and / or are hampered by the increasing demand for computing resources that hinders real-time and near-real-time applications. Summary of the Invention

[0008] A framework is disclosed for obtaining a segmentation dataset that locates one or more anatomical features in a patient's medical volumetric imaging dataset using at least one trained function. The trained function may include a residual neural network that processes a first feature vector encoding features of the medical volumetric imaging dataset and outputs a second feature vector. The trained function may also include a transformer network that processes a third feature vector obtained by combining the second feature vector with the positional encoding and generates a contextualized fourth feature vector. The trained function may further include a decoder network for generating the segmentation dataset based on the fourth feature vector. Attached Figure Description

[0009] A more complete understanding of this disclosure and its many accompanying aspects will be readily obtained when considered in conjunction with the accompanying drawings, and by referring to the following detailed description, as it becomes more readily understood.

[0010] Figure 1 The illustrations schematically depict systems including medical imaging systems and computing devices, based on various examples.

[0011] Figure 2 The diagrams illustrate processing pipelines based on various examples.

[0012] Figure 3 The illustration shows images including query locations based on various examples.

[0013] Figure 4 The diagram illustrates the sparse local descriptor sampling mesh based on various examples.

[0014] Figure 5 The illustration shows the segmentation reconstruction based on various examples.

[0015] Figure 6 The illustration shows the segmentation results based on various examples.

[0016] Figure 7 It is a flowchart based on various examples of methods.

[0017] Figure 8 It is a flowchart based on various examples of methods. Detailed Implementation

[0018] In the following description, numerous specific details, such as examples of specific components, devices, methods, etc., are set forth to provide a comprehensive understanding of implementations of this framework. However, it will be apparent to those skilled in the art that these specific details are not essential for practicing implementations of this framework. In other instances, well-known materials or methods have not been described in detail to avoid unnecessarily obscuring implementations of this framework. While this framework is readily adaptable to various modifications and alternatives, specific embodiments thereof are illustrated by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that the invention is not intended to be limited to the specific forms disclosed, but rather, it is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Furthermore, for ease of understanding, certain method steps are depicted as separate steps; however, these separately described steps should not be construed as necessarily depending on the order in which they are performed. Regardless of grammatical usage, individuals with male or female gender identities are also included in this terminology.

[0019] Unless otherwise stated, as will be apparent from the following discussion, terms such as “segment,” “generate,” “register,” “determine,” “align,” “locate,” “process,” “calculate,” “select,” “estimate,” “detect,” and “track” can refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data representing physical (e.g., electronic) quantities within the registers and memory of the computer system into other data similarly represented as physical quantities within the computer system's memory or registers or other such information storage, transmission, or display devices. Embodiments of the methods described herein can be implemented using computer software. Sequences of instructions designed to implement the methods can be compiled for execution on a wide variety of hardware platforms and interfaced with a wide variety of operating systems if written in a programming language conforming to recognized standards. Furthermore, no specific programming language is referenced in describing implementations of this framework. It will be appreciated that a wide variety of programming languages ​​can be used. Individuals with male or female gender identities are also included in the terminology, regardless of grammatical usage.

[0020] Advanced techniques are needed to provide segmented datasets for medical volumetric imaging datasets, thereby overcoming at least some of the limitations and drawbacks of the prior art mentioned above. This need is satisfied by the features of (one or more) independent claims. The features of dependent claims define the embodiments.

[0021] Based on various examples, trained functions are employed to provide segmentation data for medical volumetric imaging datasets. These trained functions can also be referred to as ML models. Specific architectures are used for ML models, depending on the specific examples.

[0022] A computer implementation method is disclosed for obtaining a segmentation dataset that locates one or more anatomical features in a patient's medical volumetric imaging dataset using at least one trained function. The at least one trained function includes a residual neural network that processes a first feature vector encoding features from the medical volumetric imaging dataset and outputs a second feature vector. The at least one trained function also includes a transformer network that processes a third feature vector obtained by combining the second feature vector with a positional encoding and generates a contextualized fourth feature vector. The at least one trained function further includes a decoder network for generating the segmentation dataset based on the fourth feature vector.

[0023] One or more anatomical features can be, for example, the lungs or any other organ. Anatomical features can include any detailed information related to human anatomy.

[0024] Medical volumetric imaging datasets of patients can be obtained through computed tomography (CT), magnetic resonance imaging (MRI), X-ray imaging, ultrasound imaging, positron emission tomography (PET), and other methods. Medical volumetric imaging datasets can capture a 3D field of view (FOV), that is, depict the patient's anatomy within a given volume. A medical volumetric imaging dataset can consist of multiple slices, such as two-dimensional (2D) slice images. 2D slice images can be assembled to form a medical volumetric imaging dataset. Medical volumetric imaging datasets may include voxel arrays.

[0025] A voxel is the 3D counterpart of a pixel. It represents a value on a regular grid in 3D space. Voxels are commonly used for the visualization and analysis of medical and scientific data. The position of a voxel can be inferred based on its position relative to other voxels. Voxels can be arranged on a 3D grid. Voxels can represent intensity, absorption, or other parameters as a function of 3D position.

[0026] Segmentation datasets can take various forms. For example, a segmentation mask can be the output of a trained function. A segmentation mask for a medical volumetric imaging dataset is a 3D array with array entries for each voxel in the dataset; each position in the array encodes the class to which the corresponding voxel belongs. This makes it possible to detect objects with voxel accuracy.

[0027] Residual Neural Networks (ResNet) are a type of Distributed Neural Network (DNN) architecture. ResNet includes residual connections, or skip connections, which perform identity mappings to connect inputs to their outputs. Residual connections allow ResNet to skip layers. Residual connections make it possible to train DNNs with a large number of layers by mitigating the vanishing gradient problem. The vanishing gradient problem is the diverging gradient problem that occurs when training a neural network using backpropagation. Backpropagation uses gradient descent to adjust the weights of the neural network. During backpropagation, residual connections are not affected by vanishing gradients because their derivatives are the identity functions. A block with residual connections is called a residual block. A typical ResNet architecture includes several residual blocks. For example, a ResNet block may include a subnetwork containing convolutions, batch normalization, an activation function, another convolution, another batch normalization, a residual connection added to the subnetwork's output, and an activation function applied after the residual connection, where the input and output dimensions are equal. Another implementation of a ResNet block may include three consecutive convolutional layers and a residual connection. Other variations of residual blocks are also possible. Using ResNet to encode features from medical volumetric imaging datasets helps to provide corresponding feature vectors that accurately capture relevant features, because the number of layers can be relatively high.

[0028] Transformer networks are DNN architectures based on attention mechanisms. Generally, a transformer network takes an input sequence of words and predicts the next word in the output sequence. In this example, a sequence of feature vectors obtained from ResNet, combined with positional encoding, is used as the sequence of word sequences. Positional encoding can take various forms, such as sine functions or learned embeddings. Positional encoding enables the transformer network to understand the relative positions of voxels or features within a medical volumetric imaging dataset. The attention mechanism determines the relative importance of each component in the sequence with respect to other components in the sequence. In a sense, the attention mechanism allows the model to move information from one embedding vector to another. The transformer can run in parallel several times through the attention mechanism. Furthermore, the transformer can contain a series of encoder and decoder layers. Transformer layers allow the network to add contextual information and capture the relationships and dependencies between all parts of the input sequence.

[0029] As an example, a volumetric imaging dataset of a patient's lungs is obtained via computed tomography (CT), magnetic resonance imaging (MRI), or any other medical imaging technique. This dataset can be embedded into a first feature vector. ResNet then processes the first feature vector, which encodes features from the medical volumetric imaging dataset, and outputs a second feature vector. For example, ResNet fuses and extracts meaningful information from the second feature vector and generates a third feature vector as input terms to the transformer network. The second feature vector can have a higher imaging resolution than the first feature vector. The third feature vector is obtained by encoding location information into the second feature vector. The third feature vector is processed by the transformer network, which uses a trained function, to generate a contextualized fourth feature vector. For example, the transformer encoder incorporates an architecture with multi-head self-attention and a feedforward network that captures contextual information between transformed terms. Residual connections can also be applied to each transformer encoder block. Based on the fourth vector, the decoder network generates a segmentation mask of the patient's lungs with high resolution and refined boundaries. The segmentation mask contains the location of one or more anatomical features (e.g., the lungs) in, for example, a medical volumetric image dataset of a patient's CT scan. Therefore, by triggering inference on at least one trained function, a segmentation dataset of one or more anatomical features in a medical volumetric imaging dataset of a localized patient is obtained.

[0030] In some examples, the computer implementation method may further include generating multiple local descriptors for a medical volumetric imaging dataset at multiple query locations, each local descriptor indicating voxel values ​​of multiple voxels in a corresponding sampled region surrounding the respective query location. The method may also include processing each local descriptor in at least one trained function to obtain a corresponding local segmentation dataset for at least a portion of the respective region, and aggregating the local segmentation datasets obtained for the multiple query locations to obtain a segmentation dataset.

[0031] Local descriptors provide imaging information about a point of interest or the area surrounding a query location in a medical volumetric imaging dataset. Local descriptors capture features from a specific region or point within the dataset. A local descriptor can be implemented as a vector containing voxel values ​​of some voxels in the neighborhood of the query location. The local descriptor encodes sufficient structural information to achieve an effective match in the target image. The spatial region surrounding the query location can be referred to as the sampling region.

[0032] Based on various examples, local descriptors can sparsely sample the sampling region. That is, a local descriptor may not include every voxel within a specific sampling region, but rather may include a subset of voxels within the sampling region. Sparse sampling means that, given the total number of voxels constituting the medical volumetric image in the corresponding neighborhood of the query location, only a small fraction of these voxels are used during the sampling process. Therefore, the dimensionality of the local descriptor is reduced, further facilitating efficient processing in the training function. Simultaneously, by appropriately adjusting the sparse sampling scheme, relevant features can be preserved in the descriptor.

[0033] For example, the query location could be a lung nodule, and the sampling region could be the nearest neighbor region surrounding the lung nodule with a predefined range. Local descriptors are generated around the query location and passed to a trained function, where they are processed by a ResNet, a transformer network, and a decoder to generate a local segmentation dataset. This process can be repeated for each local descriptor obtained by querying multiple query locations to obtain multiple local segmentation datasets for a medical volumetric imaging dataset. Multiple local segmentation datasets can be aggregated to obtain a segmentation dataset for a patient's medical volumetric imaging dataset.

[0034] The query location can be any location in the image. A query location can be, for example, any grid point of a query grid. For instance, an image can be divided into 10mm grid blocks of a query grid, and each grid point of the query grid can be a query location. For instance, multiple query locations can be set according to a predefined scheme. For instance, the query location can be determined based on the query grid. For instance, the query location can be positioned at the center point of the query grid. The query location can be positioned at the center of a unit cell of the query grid.

[0035] The query grid can be a regular Cartesian grid. Alternatively, a non-Cartesian query grid can also be used. For example, a query grid with an adaptive grid point density can be adopted based on the structural information included in the medical volumetric imaging dataset. Therefore, as an alternative to or appended to descriptor sparse sampling, an adaptive query grid can be used at each query point to incorporate the medical volumetric imaging dataset and an additional sparse layer.

[0036] Based on various examples, query grids can be used to query medical volumetric imaging datasets at uniform intervals. For example, a query grid can cover the entire volumetric imaging dataset and be uniformly spaced. The grid distance of a query grid defines the distance between two adjacent edges / grid lines of the query grid. For example, a 2D query grid with a grid distance of 10 mm can be used to cover 2D slices of a medical volumetric imaging dataset. For example, by querying the center of each query grid cell, a local descriptor is obtained for each query location.

[0037] According to the example, the computer implementation method can use sparse sampling to obtain a corresponding local descriptor for each of at least one query location in a medical volumetric imaging dataset. Each local descriptor provides a 2D array of voxel values ​​at grid points of one or more corresponding sparse local descriptor sampling grids based on the location of the corresponding query point, wherein each first feature vector is generated by encoding the corresponding descriptor. Thus, in other words, for each query location, sampling points for the descriptor are determined using one or more corresponding local descriptor sampling grids.

[0038] As will be understood, while the query position can be set according to the query grid, additionally, the sparse sampling used to determine the corresponding descriptor at any given query position can be performed according to the local descriptor sampling grid. Therefore, while the query grid is at the query position level, the local descriptor sampling grid is at the descriptor level. Different grids can be used for query and descriptor sampling respectively.

[0039] Sparse sampling can use one or more sparse local descriptor sampling grids with different grid sizes and resolutions or grid spacings. The different grid sizes of the sparse local descriptor sampling grids determine the number of distributed sampling points per descriptor. For example, a 7x7x7 sparse local descriptor sampling grid has seven sampling points or sparse grid points in each dimension. The resolution or grid spacing of the sparse local descriptor sampling grid determines the distance between two corresponding sampling points. In other words, different grid spacings or resolutions of the sparse local descriptor sampling grids result in different distances between distributed sampling points in the corresponding local descriptor. The corresponding grid spacing or resolution of the sparse local descriptor sampling grid is defined as the distance between two neighboring nodes of the corresponding sparse grid, where the corresponding node is defined as the intersection of three sparse grid lines forming a sampling point. For example, a sparse grid with a resolution of 2 mm and a grid size of 7x7x7 has seven sampling points in each dimension, and each sampling point is 2 mm apart. The grid size and resolution of the sparse local descriptor sampling grid can determine the sampling area of ​​the corresponding descriptor at the corresponding query location.

[0040] According to various examples, sparse sampling uses multiple sparse local descriptor sampling grids, such as 2D and / or 3D sparse local descriptor sampling grids with different grid sizes and / or different resolutions. For example, a high-resolution sparse local descriptor sampling grid with an 8x8 grid size and a 2 mm resolution and a low-resolution sparse local descriptor sampling grid with a 5x5 grid size and a 10 mm resolution can be used. By forming descriptors using multiple sparse local descriptor sampling grids (e.g., coarse and fine grids) preferably with different dimensions and / or different resolutions, the descriptors—although obtained from sparse sampling of a medical volumetric imaging dataset—can capture meaningful features of different dimensions and / or different resolutions. For example, relatively large-scale features are better captured by relatively low-resolution sparse local descriptor sampling grids, while small features are best captured by relatively high-resolution sparse local descriptor sampling grids. Meanwhile, since fine features tend to be localized, i.e., they tend to have a relatively limited overall range compared to coarse features, the number of grid points can be approximately the same for sparse local descriptor sampling grids with high and low resolutions.

[0041] Sparse sampling can utilize a coarse sparse local descriptor sampling grid that defines multiple coarse grid points and multiple fine sparse local descriptor sampling grids centered around two or more coarse grid points. For example, a 7x7x7 coarse sparse local descriptor sampling grid with a resolution of 64 mm can include 7 coarse grid points in 3D. For example, at each of the coarse grid points, a corresponding 5x5x5 fine sparse local descriptor sampling grid with resolutions of 3 mm and 5 mm can be set around it. The coarse and / or fine sparse local descriptor sampling grids can be 2D and / or 3D grids with different grid sizes and grid spacings. Therefore, sparse sampling can follow a hierarchical sparse sampling strategy and obtain hierarchical information. This multi-level descriptor sampling can effectively capture both coarse and fine features, even if these fine features are off-target but within the corresponding sampling region.

[0042] One or more predefined sparse local descriptor sampling meshes can include multiple 3D sparse local descriptor sampling meshes with multiple mesh sizes. For example, a sparse local descriptor sampling mesh can include a 3D sparse local descriptor sampling mesh with a 9x9x9 mesh size and another sparse local descriptor sampling mesh with a 7x7 mesh size. This has the effect of increasing the granularity of sparse sampling.

[0043] Based on the example, the mesh can be sampled using three 2D sparse local descriptors with three 27×27 orthogonal planes having a resolution of 4 mm.

[0044] For example, from coarse to fine, six 3D 9x9x9 sparse local descriptor sampling grids with resolutions of 2, 3, 5, 12, 28, and 64 mm can be used. This produces 9x9x9x9 dimensional local descriptors. For example, a local descriptor is obtained by querying the center of the query grid. In this way, the descriptor contains information about the voxels surrounding the query point from both the local and global regions.

[0045] According to the example, all sampled regions of multiple query locations acquired together sample at least 50% of the field of view of the medical volume imaging dataset, optionally at least 80%.

[0046] Therefore, sampling is essentially performed on the entire medical volumetric imaging dataset. This enables whole-body multi-organ segmentation across the entire medical volumetric imaging dataset. Sparse sampling, such as at the query grid level and / or at the descriptor level, facilitates this extensive processing across essentially the entire field of view of the medical volumetric imaging dataset. This reduces the computational burden required in the trained function to handle such a large amount of data.

[0047] Depending on the examples, the sampled regions for nearest neighbor query locations can be non-overlapping or partially overlapping. For example, when using overlapping sampled regions associated with nearest neighbor query locations, advanced logic can be used when merging the corresponding segmentation datasets output by at least one trained function. For instance, the overlap between segmentation masks can be computed. Duplicates can be removed. It can be determined whether certain segmented objects belong to the same organ / instance or segment different instances. On the other hand, by using overlapping sampled regions, the computational resources required to process medical volumetric imaging datasets increase because certain regions in the field of view of a medical volumetric imaging dataset are processed at least twice.

[0048] At least one trained function includes an encoder network comprising multiple 2D convolutional layers. The convolutional layers reduce the dimensionality of the local descriptors. For example, a local descriptor obtained through a sparse sampling method can be a 2D array of voxel values. For instance, the 2D local descriptors are processed by the convolutional layers to generate a first feature vector with only a single dimension. This one-dimensional first feature vector is then further processed by the trained function.

[0049] As illustrated in the example, the transformer network in the computer implementation method is a residual transformer network, which includes multiple multi-head self-attention layers and skip connections across two or more of the multi-head self-attention layers. Skip connections allow the network to skip layers that do not contribute to better results. Furthermore, skip connections stabilize the training process by preventing vanishing gradients.

[0050] Based on the example, a trained function is trained on the generated descriptors. For example, several medical volumetric imaging datasets can be used, divided into training and test datasets containing images of different organ classes. The generated descriptors are assembled from random locations both globally and for each organ class to achieve balanced sampling. Segmentation labels can be obtained from a 5x5x5 locally sparse local descriptor sampling grid of the center voxel of the mask. For example, multiple descriptors can be used for each training image, with 10% derived from the balanced set. For example, a random test subset containing one hundred subjects can be selected to generate uniformly sampled descriptors to evaluate segmentation performance across the entire medical volumetric imaging dataset of the patient. Once the function is successfully trained, the medical volumetric imaging dataset can be systematically queried at uniform intervals, such as a 10 mm query grid, to reconstruct the segmentation of the entire volumetric imaging dataset of the patient. Compared to organ classifiers that require querying every location in the image or utilizing edge refinement, the trained function can return voxel-by-voxel segmentation labels in 10 mm query grid blocks, thus providing high-resolution segmentation predictions within a single query in real time. Therefore, the trained function is able to capture spatial dependencies in the descriptor and provide high-resolution images with refined boundaries and sharp edges.

[0051] According to the example, the medical volumetric imaging dataset is a whole-body volumetric imaging dataset, in which the segmentation dataset is obtained from at least 80% of the whole-body volumetric imaging dataset and locates multiple organs of the patient.

[0052] According to the example, the trained function consists of multiple modules trained end-to-end.

[0053] The proposed computer implementation method can successfully reconstruct meaningful full-volume segmentation results with high accuracy and less time consumption compared with existing solutions, despite being trained using only random sparse samples, and has fine edge details and clear organ boundaries.

[0054] Some examples of this disclosure generally provide multiple circuits or other electrical devices. All references to circuits or other electrical devices and the functions provided by each device are not intended to limit the scope to what is illustrated and described herein. While specific labels may be assigned to the various circuits or other electrical devices disclosed, such labels are not intended to limit the scope of operation of the circuits and other electrical devices. Such circuits and other electrical devices may be combined and / or separated from each other in any way based on a particular type of desired electrical implementation. It should be understood that any computer system, circuit, or other electrical device disclosed herein may include any number of microcontrollers, graphics processing units (GPUs), tensor processing units (TPUs), integrated circuits such as application-specific integrated circuits or field-programmable gate arrays (FPGAs), memory devices (e.g., FLASH, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other suitable variations thereof), and software that cooperates to perform one or more of the operations disclosed herein. A computer or computing system may include at least one memory device and one or more computer processors that load program code from at least one memory device and execute the program code to perform one or more of the operations disclosed herein. In addition, any one or more of the electrical equipment may be configured to execute machine-executable program code or instructions embodied in a non-transitory computer-readable medium programmed to perform any number of disclosed functions.

[0055] In the following description, embodiments will be described in detail with reference to the accompanying drawings. It should be understood that the following description of the embodiments is not limiting. The scope of the invention is not intended to be limited by the embodiments or drawings described below, which are merely illustrative.

[0056] The accompanying drawings should be considered schematic representations, and the elements illustrated in the drawings are not necessarily shown to scale. Rather, the various elements are shown such that their function and general purpose will be obvious to those skilled in the art. Any connection or coupling between functional blocks, devices, components, or other physical or functional units shown in the drawings or described herein may also be implemented by indirect connection or coupling. Coupling between components may also be established via wireless connection. Functional blocks may be implemented in hardware, firmware, software, or a combination thereof.

[0057] Figure 1The illustration depicts a system 100 according to various examples. System 100 includes a medical imaging device 105 and a computing device 110. The medical imaging device 105 may be, for example, a magnetic resonance imaging scanner or a computed tomography scanner. The medical imaging device 105 can perform scans of a patient to obtain a medical volumetric imaging dataset of the patient. The medical volumetric imaging dataset depicts anatomical features, such as the patient's lungs, heart, or any other organ. The medical volumetric dataset can be labeled as a 3D image of the patient.

[0058] Medical imaging device 105 is connected to computing device 110. Computing device 110 includes processor 111 and memory 112. Computing device 110 also includes a human-machine interface (HMI) 113 and a communication interface 114. Medical volumetric imaging datasets of medical imaging device 105 can be visualized via HMI 113. Processor 111 can load program code stored in memory 112. When loading and executing program code, processor 111 can perform techniques disclosed herein, such as using trained functions or obtaining segmentation datasets. Segmentation datasets can be visualized via HMI 113. Processor 111 can communicate with medical imaging device 105 via communication interface 114 to obtain a patient's medical volumetric imaging dataset. For this purpose, processor 111 can provide control commands to medical imaging device 105 to trigger image acquisition. Processor 111 can then obtain the medical volumetric imaging dataset from medical imaging device 105. As a general rule, processor 111 can not only obtain medical volumetric imaging datasets from medical imaging device 105, but can also alternatively load medical volumetric imaging datasets from image archives or another imaging database.

[0059] Figure 2 The diagram illustrates a processing pipeline 200 based on various examples.

[0060] At box 2005, query the input at a specific query location, such as a patient's volumetric imaging dataset. Note that this query can be repeated multiple times for multiple query locations. Figure 2 (The dashed loop in the middle).

[0061] The local region surrounding a given query location defines the corresponding local window / sampling area for obtaining the corresponding segmented dataset.

[0062] As an example, 2D slices of a volumetric imaging dataset can be covered by a 2D query grid. For example, in Figure 3 In Figure 310, the query location of image 305 covered by a 2D query grid is shown. In various examples, the center of each cell of the query grid can be queried to obtain a locally segmented dataset for each cell of the query grid. Alternatively, each grid location of the query grid can form a query location.

[0063] Although Figure 3 The illustration shows a specific implementation of the query grid; however, as a general rule, in various disclosed examples, the query grid can have different grid spacings. For example, the query grid can have grid spacings of 5, 10, and 15 mm. The query grid can be as follows: Figure 3 The example shown is a 2D or 3D mesh. Query meshes can be uniformly spaced or have variable mesh spacing. For example, structure-aware variable mesh spacing can be used to introduce sampling sparsity at the query level.

[0064] Refer again Figure 2 In box 2010, a descriptor is determined for the corresponding query location. Here, sparse sampling is employed. This means that the corresponding sampling region associated with the corresponding query location is sparsely sampled. For this purpose, one or more sparse local descriptor sampling grids can be used. The local descriptor indicates the voxel values ​​in the corresponding sampling region surrounding the corresponding query location. The local descriptor can provide a 2D array of voxel values ​​at grid points of one or more sparse local descriptor sampling grids.

[0065] At each query location, at least one sparse local descriptor sampling grid defines the corresponding sampling region for that query location, such as... Figure 4 As shown in the image. Figure 4 The illustration shows three 3D sparse local descriptor sampling grids 504, 505, and 506 with different grid spacings. Therefore, as... Figure 4 As shown, sparse sampling defines multiple sparse local descriptor sampling grids 504, 505, and 506 with different grid spacings. These different grid spacings determine different distances D4, D5, and D6 between sampling points 507 and 508 in the medical volumetric imaging dataset. (Continue to refer to...) Figure 4Cubes (or cuboids) are designated by reference numerals 501, 502, and 503. Cubes 501, 502, and 503 are nested within each other, with the smallest cube 501 containing, for example, the query position 310 at its center. Each cube contains a corresponding sparse local descriptor sampling grid. Cube 501 contains a sparse local descriptor sampling grid 504, cube 502 contains a sparse local descriptor sampling grid 505, and cube 503 contains a sparse local descriptor sampling grid 506. The sparse local descriptor sampling grids 504, 505, and 506 may have regular spacing; they may be Cartesian grids. For illustrative purposes, the sparse local descriptor sampling grids 504, 505, and 506 are shown only in two dimensions (i.e., in the plane of the paper) and only partially. The sparse grid spacing is defined as the distance between two neighboring grid points of the corresponding sparse grid. Any grid point is defined as the intersection of three sparse grid lines. As shown in more detail for sparse local descriptor sampling grid 506, two grid points 507 and 508 are separated by a distance D6. Similarly, the sparse grid spacing of sparse local descriptor sampling grid 504 is defined by a distance D4, and the grid spacing of sparse local descriptor sampling grid 505 is defined by a distance D5. The sparse local descriptor sampling grid contains one or more sampling points represented by points 507 and 508. Voxels of the medical volumetric imaging dataset can be located at points 507 and 508 and around the query location 310.

[0066] At least one predefined sparse local descriptor sampling mesh may include, for example, one or more 3D and / or one or more 2D sparse local descriptor sampling meshes. Various sparse local descriptor sampling meshes may have multiple mesh sizes. For example, the mesh size of a sparse local descriptor sampling mesh may be 9x9x9, with resolutions of 2, 3, 5, 12, 28, and 64 mm.

[0067] For example, sparse local descriptor sampling mesh 505 can be a 2D sparse local descriptor sampling mesh with a mesh size of 9x9 and a resolution of 5 mm, and sparse local descriptor sampling mesh 506 can be a 3D sparse local descriptor sampling mesh with a mesh size of 9x9x9 and a resolution of 12 mm. In another example, sparse local descriptor sampling mesh 504 can be a 3D sparse mesh with a mesh size of 9x9x9 and a resolution of 5 mm, sparse local descriptor sampling mesh 505 can be a 3D sparse local descriptor sampling mesh with a mesh size of 9x9x9 and a resolution of 12 mm, and sparse local descriptor sampling mesh 506 can be a 3D sparse local descriptor sampling mesh with a mesh size of 5x5x5 and a resolution of 28 mm.

[0068] In various examples, at least one predefined sparse local descriptor sampling grid includes a coarse sparse local descriptor sampling grid defining multiple coarse grid points, and also includes multiple fine grids centered on two or more of the coarse grid points. For example, a 7x7x7 coarse grid with a resolution of 64 mm can contain one or more 5x5x5 fine grids with a resolution of 5 mm. For example, each of the seven coarse grid points in one dimension of the coarse sparse local descriptor sampling grid representing a sampling point is surrounded by five fine grid points of a fine sparse local descriptor sampling grid that also represents a sampling point in one dimension. The fine grid points are centered on the coarse grid points. In another example, a 7x7x7 coarse grid with a resolution of 64 mm can contain 5x5x5 fine grid points with a resolution of 5 mm and another 5x5x5 fine sparse local descriptor sampling grid with a resolution of 2 mm. The number of fine sparse local descriptor sampling grids centered on the coarse sparse local descriptor sampling grid points is unlimited. Thus, hierarchical information is obtained, and a coarse-to-fine strategy can be utilized.

[0069] Refer again Figure 2 At box 2010, generate a local descriptor at the query location or query point.

[0070] Then, for each local descriptor, a first feature vector is generated by encoding the corresponding descriptor through projection and flattening. The first feature vector encodes features from the medical volumetric imaging dataset. The dimensionality of the local descriptors is reduced by a trained function. For example, the dimensionality can be reduced by convolutional layers of a trained function. For instance, the trained function may include an encoder network comprising multiple 2D convolutional layers to reduce the dimensionality of the local descriptors.

[0071] At box 2020, the first feature vector encoding the local descriptor is processed by ResNet. ResNet fuses and extracts meaningful information from the first feature vector and outputs a second feature vector. ResNet can contain multiple convolutional layers and / or linear layers and / or normalization layers, as well as one or more skip connections. Skip connections avoid the vanishing gradient problem and stabilize the network. Due to the ResNet structure, the trained function can extract high-resolution features from medical volumetric imaging datasets. The third feature vector is obtained by combining the second feature vector with positional encoding.

[0072] At box 2030, the third feature vector represents the input token and is processed by a transformer architecture of a trained function. This transformer contains layers with a standard architecture including multi-head self-attention and feedforward blocks that capture contextual information. In various examples, the transformer network is a residual transformer network, which includes multiple multi-head self-attention layers and skip connections across two or more of these layers. Skip connections stabilize the transformer network by preventing gradient vanishing during training. The transformer generates a fourth feature vector. The transformer layers have the effect of capturing spatial dependencies between descriptors, resulting in sharp edges and refined boundaries.

[0073] At box 2035, the decoder network generates a segmentation mask based on the fourth feature vector. The segmentation mask is generated within the corresponding sampling region associated with the current query location.

[0074] The decoder network includes, for example, two convolutional layers to match the dimension of the segmentation mask output. The sampled region can provide a segmentation mask for the queried sampled region in the same dimension as the input that provides local fine details.

[0075] You can repeat the query for multiple locations (dashed arrows). Figure 2 The process 200 shown yields a segmentation dataset that locates one or more anatomical features within a patient's medical volumetric imaging dataset. For example, a 10 mm query grid can be used, and queries can be performed at the center of each grid cell. Thus, a segmentation mask is obtained within the sampling region at each query location. By aggregating the local segmentation datasets, a segmentation dataset for the entire image is obtained. In this way, a segmentation dataset for the entire input image is obtained. This is in... Figure 5 The diagram is shown in the image. Specifically, Figure 5 Image 405 is shown, which is a horizontal slice of a computed tomography scan with a grid covering image 405 (see image 405). Figure 3 Image 415 shows the result of querying each query grid center 410 of image 405. The corresponding sampling mask is shown. Multiple organs are segmented using the corresponding sampling mask, thus achieving organ-level instance segmentation.

[0076] Figure 6 The illustrations depict segmentation datasets based on various examples. Image 605 shows a horizontal slice of a patient's 3D computed tomography scan, and image 610 shows a vertical slice of a patient's 3D computed tomography scan. Images 606 and 611 show segmentation training samples used to train at least one trained function. Images 607 and 612 show segmentation results from the proposed computer implementation method. According to the examples, images 605 and 610 are based on... Figure 2The processing pipeline 200 queries and processes input images at multiple query locations. Images 607 and 612 are the outputs of at least one trained function illustrating a segmentation dataset of patients. The computer-implemented method is able to successfully reconstruct meaningful full-volume segmentation results with high accuracy, fine edge details, and clear organ boundaries, despite being trained using only random sparse samples.

[0077] Figure 7 It is a flowchart based on various examples of methods. Figure 7 The method can be executed by a computing device. More specifically, when program code is loaded from memory and executed, Figure 7 The method can be executed by the processor. For example, Figure 7 At least a portion of the method may be made by the processor 111 of the computing device 110 (see...) Figure 1 )implement.

[0078] At box 7005, the training function obtains a segmentation dataset from a patient's medical volumetric imaging dataset. For example, a training dataset of tens of thousands of images obtained from computed tomography scans of different patients with different organ classes can be used to train the function. The training dataset can be split into training samples and validation training samples. For training, descriptors are generated from both global and random locations from each organ class to achieve balanced sampling. Segmentation labels can be derived from a 5x5x5 local grid of the center voxel of the mask. For example, in Figure 6 The segmentation labels are shown in the figure, where 606 and 611 are two segmentation labels. Furthermore, each training sample can be randomly sampled with one thousand or more descriptors. A random validation subset containing hundreds of validation samples can be selected to generate uniformly sampled sparse descriptors to evaluate the full-volume segmentation performance. Therefore, the trained function is able to successfully reconstruct meaningful full-volume segmentation results with high accuracy, despite being trained using only randomized sparse descriptors. On a standard consumer-grade CPU, the model achieves an average runtime of several seconds on a full volume segmentation of a medical imaging volume dataset.

[0079] Based on various examples, the trained function comprises multiple modules trained end-to-end.

[0080] At box 7010, inference of the trained function is triggered. For inference, the trained function can be deployed locally at the edge computing hardware. At box 7010, execution can be performed... Figure 2 The data processing pipeline.

[0081] Figure 8 It is a flowchart based on various examples of methods. Figure 8 The method can be executed by a computing device. More specifically, when program code is loaded from memory and executed, Figure 8 The method can be executed by the processor. For example, Figure 8 At least a portion of the method may be made by the processor 111 of the computing device 110 (see...) Figure 1 )implement. Figure 8 The method corresponds to the inference of the trained function (see...). Figure 7 (Frame 7010).

[0082] At box 8005, obtain the medical imaging dataset. The medical imaging dataset can be obtained from a patient's computed tomography scan or another 3D medical imaging device (see [link]). Figure 1 Medical imaging datasets can be obtained from a data repository. These datasets may contain, for example, a patient's organs or other anatomical features.

[0083] At box 8010, a pre-trained function is used to process the medical imaging dataset obtained at box 8005.

[0084] Box 8010 contains sub-box 8015, where sparse sampling is applied. Sparse sampling involves generating local descriptors for each of multiple query locations. Each local descriptor includes, for example, one or more 2D arrays, each array containing a sampling grid around the corresponding 2D or 3D sparse local descriptor (see...). Figure 4 The voxel value at the query location () is given. Due to sparse sampling at different resolutions, the local descriptor obtains information from both the global and local regions. The local descriptor obtained at the query location through sparse sampling is further processed by a trained function, such as... Figure 2 As described in [the text].

[0085] At box 8020, the pre-trained function outputs a segmentation dataset containing one or more anatomical features, for example, as previously shown in... Figure 5 and Figure 6 As shown in the example. According to various examples, a medical volumetric imaging dataset is a whole-body volumetric imaging dataset, wherein the segmentation dataset is obtained for at least 80% of the whole-body volumetric imaging dataset, and multiple organs of the patient are located.

[0086] For illustration, various examples of using query grids that define query points have been presented above, with local descriptors determined for each query point. The use of query grids is often optional. For example, query points can also be randomly distributed across the field of view of a medical volumetric imaging dataset.

[0087] Although the invention has been shown and described with respect to certain preferred embodiments, equivalents and modifications will occur to those skilled in the art upon reading and understanding the specification. The invention includes all such equivalents and modifications and is limited only by the scope of the appended claims.

Claims

1. A computer implementation method, comprising: By using at least one trained function, a segmentation dataset of one or more anatomical features in a medical volumetric imaging dataset of a patient can be obtained. The at least one trained function includes a residual neural network, which processes a first feature vector encoding features of the medical volumetric imaging dataset and outputs a second feature vector. The at least one trained function further includes a transformer network that processes a third feature vector obtained by combining a second feature vector with a positional encoding, and generates a contextualized fourth feature vector. The at least one trained function further includes a decoder network for generating a segmented dataset based on a fourth feature vector.

2. The computer implementation method according to claim 1, further comprising: Multiple local descriptors of the medical volume imaging dataset are generated at multiple query locations, each local descriptor indicating the voxel values ​​of multiple voxels in a corresponding sampling region surrounding the corresponding query location; Each local descriptor is processed in the at least one trained function to obtain a corresponding local segmentation dataset for at least a portion of the corresponding sampled region; as well as The segmented dataset is obtained by aggregating the local segmented datasets obtained for the multiple query locations.

3. The computer implementation method according to claim 2, in, The sampling regions at the plurality of query locations sample at least 50% of the field of view of the medical volumetric imaging dataset.

4. The computer implementation method according to claim 3, in, The sampling regions at the plurality of query locations sample at least 80% of the field of view of the medical volumetric imaging dataset.

5. The computer implementation method according to claim 2, in, The sampling regions of the nearest neighbor query locations do not overlap or partially overlap.

6. The computer implementation method according to claim 1, wherein, The at least one trained function includes an encoder network, which includes multiple two-dimensional convolutional layers.

7. The computer implementation method according to claim 1, further comprising: For each of at least one query location in the medical volume imaging dataset, sparse sampling is used to obtain a corresponding local descriptor, which provides a two-dimensional array of voxel values ​​at grid points of one or more predefined sparse grids located based on the corresponding query location.

8. The computer implementation method according to claim 7, wherein, Each first feature vector is generated by encoding the corresponding descriptor.

9. The computer implementation method according to claim 7, in, At least one predefined sparse mesh includes a coarse mesh defining a plurality of coarse mesh points, and further includes a plurality of fine meshes centered at two or more of the coarse mesh points.

10. The computer implementation method according to claim 9, in, The at least one predefined sparse mesh includes at least one two-dimensional predefined sparse mesh, and further includes at least one three-dimensional predefined sparse mesh.

11. The computer implementation method according to claim 9, wherein, The at least one predefined sparse mesh comprises multiple three-dimensional predefined sparse meshes of multiple mesh sizes.

12. The computer implementation method according to claim 1, wherein, The transformer network is a residual transformer network, which includes multiple multi-head self-attention layers and skip connections across two or more of the multi-head self-attention layers.

13. The computer implementation method according to claim 1, wherein, The medical volumetric imaging dataset is a whole-body volumetric imaging dataset, wherein the segmentation dataset is obtained for at least 80% of the whole-body volumetric imaging dataset and locates multiple organs of the patient.

14. The computer implementation method according to claim 1, wherein, The trained function comprises multiple modules trained end-to-end.

15. A computer system comprising: At least one memory device; as well as One or more computer processors, the one or more computer processors loading program code from the at least one memory device and executing the program code, the execution of the program code causing the one or more computer processors to: By using at least one trained function, a segmentation dataset of one or more anatomical features in a medical volumetric imaging dataset of a patient can be obtained. The at least one trained function includes a residual neural network, which processes a first feature vector encoding features of the medical volumetric imaging dataset and outputs a second feature vector. The at least one trained function further includes a transformer network that processes a third feature vector obtained by combining a second feature vector with a positional encoding, and generates a contextualized fourth feature vector. The at least one trained function further includes a decoder network for generating a segmented dataset based on a fourth feature vector.

16. The computer system according to claim 15, wherein, The execution of the program code causes the one or more computer processors to: Multiple local descriptors for the medical volumetric imaging dataset are generated at multiple query locations. Each local descriptor indicates the voxel values ​​of multiple voxels in a corresponding sampling region surrounding the corresponding query location. Each local descriptor is processed in the at least one trained function to obtain a corresponding local segmentation dataset for at least a portion of the corresponding sampled region, and The segmented dataset is obtained by aggregating the local segmented datasets obtained for the multiple query locations.

17. The computer system according to claim 16, in, The sampling regions at the plurality of query locations sample at least 50% of the field of view of the medical volumetric imaging dataset.

18. The computer system according to claim 16, in, The sampling regions of the nearest neighbor query locations do not overlap or partially overlap.

19. The computer system according to claim 15, wherein, The execution of the program code causes the one or more computer processors to: For each of at least one query location in the medical volume imaging dataset, sparse sampling is used to obtain a corresponding local descriptor, which provides a two-dimensional array of voxel values ​​at grid points of one or more predefined sparse grids located based on the corresponding query location.

20. One or more non-transitory computer-readable media embodying machine-executable instructions for performing operations, said operations including: By using at least one trained function, a segmentation dataset of one or more anatomical features in a medical volumetric imaging dataset of a patient can be obtained. The at least one trained function includes a residual neural network, which processes a first feature vector encoding features of the medical volumetric imaging dataset and outputs a second feature vector. The at least one trained function further includes a transformer network that processes a third feature vector obtained by combining a second feature vector with a positional encoding, and generates a contextualized fourth feature vector. The at least one trained function further includes a decoder network for generating a segmented dataset based on a fourth feature vector.

Images