Method for identifying organ types in volumetric medical images

Abstract

To provide a method, apparatus, and system for identifying a type of organ (309) in a volumetric medical image (305).SOLUTION: In one aspect, the method comprises: a) receiving the volumetric medical image (305), the volumetric medical image including at least one organ (309) or portion thereof; b) receiving a single point of interest (310) within the volumetric medical image (305); c) sampling voxels (306, 308) from the volumetric medical image (305), where at least one voxel (307) is skipped between the two sampled voxels (306, 308); and d) identifying the type of organ (309) at the single point of interest (310) by applying a trained classifier to the sampled voxels (306, 308).SELECTED DRAWING: Figure 3A

Description

[Technical Field]

[0001] The present invention relates to a computer-implemented method, apparatus, system, computer program product, and computer-readable medium for identifying organ types in volumetric medical images. [Background technology]

[0002] One of the core problems in medical imaging is segmentation, which is used to identify organ boundaries, visualize displays, or calculate volumes. One prominent use case is organ identification during location searching. However, fully segmenting all organs for this purpose would be computationally intensive, as a 3D image can contain billions of voxels (or volume elements). Also, full organ segmentation is not required in most cases to identify organs at locations of interest.

[0003] Another method for identifying organ position information is landmarking (or landmarks), however landmarks do not provide information on the inside and outside of the selected location due to their coarse level of representation.

[0004] Recent work on fine-grained (or coarse) organ segmentation has reduced the computational burden to a few seconds, but such segmentation techniques still require additional hardware to implement, making them impractical in many settings.

[0005] It is possible to pre-compute the segmentation mask before taking the radiology readout, but this process complicates database design, authentication, and communication protocols.

[0006] Non-patent document 1 and non-patent document 2 disclose prior art. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Zhang, Fan, Yu Wang, and Hua Yang, "An Efficient Context-Aware Network for Abdominal Multiple Organ Segmentation," arXiv (archive) preprint, arXiv:2109.10601 (2021) [Non-patent document 2] Yan, Zhennan et al., "Body Part Recognition Using Multi-Stage Deep Learning," International Conference on Medical Image Processing, Springer-Cham (2015) Summary of the Invention [Problem to be solved by the invention]

[0008] It is therefore an object of the present invention to provide a method, an apparatus (or device) and a system that allows for improved organ identification. [Means for solving the problem]

[0009] In a first aspect of the present invention, there is provided a computer-implemented method for identifying organ types in volumetric medical images, the method comprising: a) receiving a volumetric medical image including at least one organ or part of an organ; b) receiving a single point of interest in the volumetric medical image; c) sampling voxels from the volumetric medical image, skipping at least one voxel between two sampled voxels; d) applying a trained classifier to the sampled voxels to identify the organ type at the single point of interest.

[0010] Advantageously, at least one voxel is skipped between two sampled voxels. This reduces the amount of data that needs to be processed by the trained classifier to identify the organ type, thereby reducing computation time and resources. The inventors have found that using this method, the amount of data processed by the trained classifier is reduced, yet the organ type can still be reliably identified. One reason for this is that the sampled voxels (which may also be referred to herein as "descriptors") correspond to a larger field of view and therefore take into account neighborhood information compared to when all voxels in a smaller subvolume are sampled.

[0011] An organ (or organ system) can be understood as a collection of tissues joined together in a structural unit that performs a common function. The organ may be a human organ. The organ may be, for example, any of the intestines, skeleton, kidneys, gallbladder, liver, muscles, arteries, heart, larynx, pharynx, brain, lymph nodes, lungs, bone marrow, spleen, stomach, veins, pancreas, and bladder.

[0012] Volumetric medical images may be captured and received by a medical imaging device (or imaging device). Non-limiting examples of medical imaging devices include magnetic resonance imaging devices, computed tomography devices, x-ray imaging devices, ultrasound imaging devices, etc. Volumetric medical images may be three-dimensional (3D) and / or volumetric. A volumetric medical image may be composed of multiple slices, i.e., multiple two-dimensional (2D) medical images. The 2D medical images may be captured and received by the medical imaging device. The 2D medical images may then be assembled to form the volumetric medical image.

[0013] As used herein, a voxel (volume element) is intended to represent a value in three-dimensional space, and a pixel (picture element) is intended to represent a value in two-dimensional space. Pixels or voxels may or may not have their locations, i.e., they may or may not have their coordinates explicitly encoded in their values. Alternatively, the location of a pixel or voxel may be inferred based on its relative location to other pixels or voxels (i.e., located within a data structure that constitutes a single 2D or 3D (volumetric) image). Voxels may be arranged on a 3D grid, and pixels may be arranged on a 2D grid. A 2D medical image may, for example, be in the form of an array of pixels. A volumetric medical image may include an array of voxels. The pixels of multiple 2D medical images that make up a volumetric medical image may also be referred to herein as voxels. A pixel or voxel may represent intensity, absorption or other parameters as a function of three-dimensional position and may be obtained, for example, by suitable processing of measurement signals obtained from one or more of the above medical imaging devices.

[0014] A single point of interest in the volumetric medical image may correspond to one (exactly one) pixel or voxel in three-dimensional space, or the position of said pixel or voxel. Steps a) to d) of the method may be repeated for different single points of interest, for example to identify different organs in the volumetric medical image.

[0015] Sampling a voxel may be performed by reading it from a data file, a database, (e.g., temporary) memory, or an array containing voxels. Sampling a voxel may be performed sequentially or in parallel (e.g., reading multiple voxels simultaneously). At least one voxel is skipped between two sampled voxels. This means that when viewing all voxels of a volumetric medical image in their three-dimensional relationship, at least one voxel is not sampled between two sampled voxels. For example, if a volumetric medical image includes first, second, and third voxels arranged in the same column or row, only the first and third voxels are sampled, and the second voxel is not sampled. In this case, for example, the first voxel may be sampled first, and then the third voxel may be sampled. Alternatively, the first and third voxels may be sampled in parallel. The sampled voxels may be stored in memory.

[0016] A trained classifier is applied to the sampled voxels. In particular, when a sampled voxel is read from memory, it is then processed by a trained classifier. The trained classifier can be, for example, a trained neural network.

[0017] In one embodiment, a robotic (e.g., CT or MR) scanner or other device or machine is controlled based on the identified type of organ (or organ-specific abnormality or lesion, as described below). For example, the robot may be configured to operate on the patient's body. In particular, the movement of the robot (e.g., a surgical instrument such as a scalpel) or scanner may be controlled based on the identified organ.

[0018] In one embodiment, step c) samples the voxels in a sparse and / or random manner.

[0019] "Sparse" can be understood as meaning that, when considering the total number of voxels constituting the volumetric medical image, only a small number of voxels are used in the sparse sampling. In particular, "sparse" means that in step c), less than 50%, less than 20%, or even less than 10% of the total number of voxels of the volumetric medical image are sampled.

[0020] By "random" we mean that the sampled voxels do not follow a regular pattern (although at least one voxel is skipped between two sampled voxels). The (random) pattern used in step c) may be the same for different single interest points. In some embodiments, a random or pseudo-random number generator can be used to select the voxels to be sampled in step c).

[0021] In one embodiment, step c) includes sampling the voxels at a sampling rate per unit length, area, or volume that decreases with the distance of each voxel from a single point of interest.

[0022] Thus, a field of view is obtained that is focused on a single point of interest while also taking into account distance information away from that point of interest.

[0023] In step c), a sampling model can be used, which contains information about the location of the voxels in the volumetric medical image to be sampled, thus providing the descriptors. The sampling model can, for example, be an algorithm or utilize an algorithm.

[0024] In one embodiment, the sampling rate decreases at a non-linear rate, in particular at an exponential, logarithmic or power function rate.

[0025] The inventors have found that using the sampling rates described herein can significantly reduce computation time while still reliably distinguishing organ types.

[0026] In a further embodiment, the sampled voxels represent less than 1%, preferably less than 0.1%, and more preferably less than 0.01% of the total number of voxels in the volumetric medical image.

[0027] In one embodiment, the trained classifier is a neural network, in particular a multi-layer perceptron, a convolutional neural network, a Siamese network, or a triplet network.

[0028] The difference between classification (using a trained classifier) ​​and segmentation (also typically performed using neural networks) is that classification outputs a single label (e.g., organ type) for a single interest point (e.g., a particular voxel), whereas segmentation determines a label for each voxel in the input data. Thus, training and applying classification is substantially different from training and applying segmentation.

[0029] It is possible to use a simple neural network classifier to classify organ labels using descriptors at the location of interest, but this neural network classifier is not a translation-invariant model because the descriptors change with changes in location.

[0030] As an example, the neural network can be a single-layer logistic regression, and its computation can be a softmax of a linear combination of the descriptor values ​​formulated as follows:

[0031]

number

[0032] In the above formula, "l organ " refers to the organ label, and "W l " refers to the weight vector of different organs, and "x" refers to the descriptor.

[0033] In other cases, multi-layer neural networks can be used to model non-linear relationships, for example by adding at least one hidden layer to the input descriptors and, in the case of the previous example, a softmax classifier layer.

[0034] In a further embodiment, the received volumetric medical image, or a portion of the volumetric medical image containing a single point of interest, is displayed on a graphical user interface (or GUI), and a semantic description (or semantic description data) of the identified organ type may be generated and displayed at or near the location of the single point of interest.

[0035] In this way, the user can quickly understand what type of organ is located at a single point of interest.

[0036] In one embodiment, a single point of interest is selected by the user. For example, a single point of interest may be selected using a graphical user interface and an input device, such as a pointer device, that interacts with the graphical user interface to select the single point of interest. In other embodiments, the single point of interest may be input using a keyboard, a data file, or the like.

[0037] In a further embodiment, a single point of interest may be selected by pausing a cursor manipulated by a user over a volumetric medical image or a portion thereof displayed on a graphical user interface.

[0038] In this example, classification (step d) is fast, e.g., occurs in less than 10 ms (which may be faster than the refresh rate of a screen used as part of a graphical user interface), and the organ type, e.g., semantic description, associated with a single point of interest may be displayed each time the cursor is briefly paused. Here, "paused" means that the cursor is not being moved by the user. This allows a user (e.g., a physician) to perform a fast and efficient analysis of volumetric medical images.

[0039] In a further embodiment, when a user takes a measurement on a volumetric medical image or a portion thereof, the identified type of organ is stored in a database along with the measurement taken.

[0040] Often, a user makes measurements on a volumetric medical image or a portion thereof. For example, a physician may measure the size of an organ or the size of a lesion or tumor within the organ. This measurement can be made, for example, using a graphical user interface. Obtaining measurements associated with a particular organ can make subsequent referencing it simplified and more efficient.

[0041] In a further aspect of the invention, the method further comprises: receiving a plurality of untrained classifiers for identifying organ-specific anomalies (or abnormalities); selecting one or more untrained classifiers from the plurality of classifiers based on the identified organs; training the one or more untrained classifiers using volumetric medical images; Includes.

[0042] Classifiers (e.g., neural networks) exist that are configured to identify organ types in volumetric medical images. Other classifiers (e.g., neural networks) exist that are configured to identify organ-specific anomalies. These organ-specific anomalies include, for example, nodules, lesions, and tumors. Because these organ-specific anomalies differ substantially in shape, texture, density, etc., it is preferable to train the classifiers for each organ type. Therefore, the training data used for these organ-specific classifiers must be classified. For example, a lung nodule classifier can be trained using volumetric medical images showing the lungs, a liver lesion classifier can be trained using volumetric medical images showing the liver, and so on. This method can be efficiently used to provide each classifier with the appropriate volumetric medical images.

[0043] In a further embodiment, the method is further configured to perform and / or repeat steps a)-d) for each of "N-1" single points of interest in the volumetric medical image, where "N" is less than or equal to the total number of voxels in the volumetric medical image.

[0044] It is often necessary to determine the organ type for multiple single interest points in a volumetric medical image. Therefore, the present method can be performed or repeated multiple times for different single interest points. "N" is a positive, real, and integer number. The difference between "performing" and "repeating" is that "performing" allows for parallel execution, while "repeating" requires a sequential approach. Typically, the N interest points correspond to nodes of a coarse grid applied to the volumetric medical image. In other words, the N single interest points correspond to a grid with a grid spacing greater than the spacing of voxels contained in the volumetric medical image. Preferably, the grid has a regular spacing. However, in the extreme case, when N is equal to the total number of voxels, similar results can be obtained compared to segmentation methods. However, in this method, all single interest points are evaluated independently of each other.

[0045] Preferably, steps a)-d) are performed in parallel on one or more processing devices, thus further reducing processing time.

[0046] In a second aspect of the present invention, there is provided a computer-implemented method for training the classifier (see step d) above), the method comprising: a) receiving a volumetric medical image including at least one organ or part of an organ, and at least one organ type; b) receiving a single point of interest in the volumetric medical image; c) sampling voxels from the volumetric medical image, skipping at least one voxel between two sampled voxels; d) identifying the organ type at the single interest point by applying an untrained classifier to the sampled voxels; e) comparing the identified organ type with the received organ type; f) adjusting said classifier based on the comparison in step e) to obtain a trained classifier; Includes.

[0047] The received at least one organ type can be stored in a database prior to step a). The at least one organ type can be determined prior to step a) by manually inspecting the volumetric medical images received in step a), or can be determined in other ways. Modifying the classifier according to step f) can include changing weights in the neural network of the untrained or partially trained classifier.

[0048] In a third aspect of the present invention, there is provided an apparatus (or device) for identifying organ types in volumetric medical images, the apparatus comprising: one or more processing units; a first receiving device configured to receive one or more volumetric medical images captured by the medical imaging device; a second receiving device configured to receive a single point of interest in the one or more volumetric medical images; a memory (storage device) coupled to the one or more processing devices and including modules configured to perform the steps of the method; Includes.

[0049] Each device, e.g., the processing device, the first receiving device, or the second receiving device, may be implemented in hardware and / or software. If the device is implemented in hardware, it may be implemented as a device, e.g., a computer, a processor, or part of a system (e.g., a computer system). If the device is implemented in software, it may be implemented as a computer program, a function, a routine, program code, or an executable object.

[0050] In a fourth aspect of the present invention, there is provided a system for identifying organ types in volumetric medical images, the system comprising: one or more servers; a medical imaging device coupled to the one or more servers; Includes. The one or more servers include instructions that, when executed, cause the one or more servers to perform the steps of the method.

[0051] In a fifth aspect of the present invention, there is provided a computer program product comprising machine-readable instructions which, when executed by one or more processing devices, cause the one or more processing devices to perform the steps of the method described above.

[0052] A computer program product, such as a computer program means, may be implemented, for example, as a memory card, as a USB stick, as a CD-ROM, as a DVD, or as a file downloadable from a server in a network, for example provided by transferring the file comprising the computer program product over a wireless communications network.

[0053] In a sixth aspect of the present invention, there is provided a computer-readable medium having stored thereon program code portions of a computer program, the program code portions being loadable and / or executable within a system such that when the program code portions are executed within the system they cause the system to perform the steps of the method.

[0054] Features, advantages and embodiments described with respect to the first aspect apply equally to the second and subsequent aspects, and vice versa.

[0055] It should be understood that, as used herein, "one" (or the article / A meaning one) is not used to limit a single component. Rather, unless explicitly stated, it can mean one or more components. Furthermore, it should be understood that "ordinal numbers (a, b, etc.)" (e.g., step a), step b), etc.) are not used to limit a particular order. Rather, the order can be changed as deemed appropriate by one skilled in the art.

[0056] Further possible implementations or alternative solutions of the present invention may include combinations of features described in the above or below embodiments, even if not explicitly described herein, and those skilled in the art will be able to add independent or separate aspects or features to the most basic embodiment of the present invention.

[0057] Further embodiments, features, and advantages of the present invention will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0058] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary client-server architecture implementing a system for identifying organ types in volumetric medical images. [Figure 2] FIG. 2 is a block diagram of a data processing system implementing an apparatus for identifying organ types in volumetric medical images. [Figure 3] FIG. 3 illustrates a flowchart implementing a computer-implemented method for identifying organ types in volumetric medical images. [Figure 3A] FIG. 3A is an illustration of a volumetric medical image in which some voxels are scanned while others are skipped. [Figure 4] FIG. 4 illustrates a slice of a volumetric medical image, illustrating a portion of a sampling model of location information, according to one embodiment. [Figure 5] Figure 5 illustrates the three-dimensional grids used to define the sampling model in Figure 4. [Figure 6] FIG. 6 illustrates intensity values ​​(descriptors) at the time of sampling when the sampling model of FIG. 4 is applied to a slice of the volumetric medical image illustrated in FIG. [Figure 7] FIG. 7 is a diagram illustrating a graphical user interface according to one embodiment. [Figure 8] Figure 8 illustrates the graphical user interface with measurements. [Figure 9] FIG. 9 illustrates an embodiment of a database for storing results. [Figure 10] FIG. 10 illustrates various classifiers used in a federated learning approach according to one embodiment. [Figure 11] FIG. 11 illustrates a flowchart of a method for training an organ-specific classifier according to one embodiment. [Figure 12] FIG. 12 illustrates an embodiment of a coarse liver segmentation. [Figure 13] FIG. 13 illustrates a flowchart of a computer-implemented method for training a classifier used herein. DETAILED DESCRIPTION OF THE INVENTION

[0059] DETAILED DESCRIPTION OF THE INVENTION

[0013] Embodiments of the present invention will now be described in detail. Various embodiments will be described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more embodiments. It will be apparent that embodiments of the present invention may be practiced without these specific details.

[0060] FIG. 1 is a block diagram illustrating a client-server architecture for identifying organ types in volumetric medical images. The client-server architecture 100 includes a server 101 and multiple client devices 107A-N. Each of the client devices 107A-N is connected to the server 101 via a network 105, such as a local area network (LAN), a wide area network (WAN), or Wi-Fi. In one embodiment, the server 101 is located within a cloud computing environment. As used herein, the term "cloud computing environment" refers to a processing environment that includes physical and logical resources for configurable computing (e.g., networks, servers, storage, applications, services, etc.) and data distributed across the network 105 (e.g., the Internet). A cloud computing environment provides on-demand network access to a shared pool of physical and logical resources for configurable computing. The server 101 may include a medical database 102 that includes medical images for multiple patients, the images being managed by a healthcare provider. In one embodiment, the medical database 102 includes volumetric medical images captured by, for example, an MR scanner and / or a CT scanner. The server 101 may include a module 103 configured to identify organ types in the volumetric medical images, as described below, among other things.

[0061] The client-side devices 107A-N are user-side devices used by users, such as medical professionals, e.g., radiologists, pathologists, physicians, etc. In one embodiment, the user-side devices 107A-N may be used by users to receive volumetric or 2D (two-dimensional) medical images related to a patient. The data is accessible by the users through a GUI (graphical user interface) of an end-user web application on the user-side devices 107A-N. In another embodiment, a request may be sent to the server 101 via the network 105 to access medical images related to a patient.

[0062] An imaging unit (or medical imaging device) 108 can be connected to the server 101 via a network 105. The unit 108 can be a medical imaging unit 108 capable of acquiring a plurality of volumetric medical images. The medical imaging unit 108 can be, for example, a scanner unit such as a magnetic resonance imaging unit, a computed tomography imaging unit, an X-ray fluoroscopic imaging unit, an ultrasound imaging unit, or the like.

[0063] 2 is a block diagram illustrating a data processing system 101 in which embodiments of the present invention may be implemented, such as a system 101 for identifying organ types in volumetric medical images configured to perform the processes described below. It should be understood that the server 101 is an exemplary implementation of the system of FIG. 2. In FIG. 2, the data processing system 101 includes a processing unit 201, a memory 202, a storage unit 203, an input device 204, an output device 206, a bus 205, and a network interface 104.

[0064] As used herein, processing device 201 refers to any type of computing circuitry, including, but not limited to, a microprocessor, a microcontroller, a complex set of instructions for a microprocessor to compute, a smaller set of instructions for a microprocessor to compute, a very long instruction word for a microprocessor, explicitly parallel instructions for a microprocessor to compute, a graphics processor, a digital signal processor, or any other type of processing circuitry. Processing device 101 can also include embedded controllers, such as, for example, general purpose or programmable logic devices or arrays, application specific integrated circuits, single chip computers, etc.

[0065] The memory 202 may be volatile or non-volatile memory. The memory 202 may be communicatively coupled to the processing unit 201. The processing unit 201 may execute instructions and / or code stored in the memory 202. Various computer-readable storage media may be stored in and accessible from the memory 202. The memory 202 may include any suitable component capable of storing data or machine-readable instructions, including, for example, read-only memory, random-access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, a hard drive, a removable media drive for handling compact discs (CDs), digital video disks, diskettes, magnetic tape cartridges, memory cards, etc. In this embodiment, the memory 201 includes executable modules 103 stored in the form of machine-readable instructions on any of the above storage media and in communication with the processing unit 201. When executed by the processing unit 201, the module 103 causes the processing unit 201 to identify organ types in a volumetric medical image. The method steps performed by the processing unit 201 to achieve the above functionality are described with reference to the figures below.

[0066] The storage device 203 may be a non-transitory storage medium capable of storing the medical database 102. The input device 204 may include input means such as a keypad, a touch-sensitive display, a camera (e.g., gesture-based input captured by the camera), a port capable of providing input signals such as a mouse input signal or a camera input signal. The bus 205 serves to interconnect the processor (processing device) 201, the memory 202, the storage device 203, the input device 204, the output device 206, and the network interface 104. The volumetric medical images may be read into the medical database 102, for example, via the network interface 104 or the input device 204.

[0067] Those skilled in the art will appreciate that the above hardware illustrated in Figure 1 may vary according to the particular implementation. For example, other peripheral devices such as optical disk drives, local area network (LAN) / wide area network (WAN) / wireless (e.g., Wi-Fi) adapters, graphics adapters, disk controllers, and input / output (I / O) adapters may be used in place of or in addition to the illustrated hardware. The above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0068] A data processing system 101 according to embodiments of the present disclosure may include an operating system (OS) that uses a graphical user interface (GUI). The operating system may display multiple display windows within the graphical user interface, with each display window providing an interface for a different application or for different instances of the same application. A cursor within the graphical user interface may be manipulated by a user via a pointing device. The cursor's position may be changeable and / or may be capable of generating a desired response upon the occurrence of an event, such as a mouse button click.

[0069] Any of a variety of commercially available operating systems may be used, such as any version of Microsoft Windows®, a product of Microsoft Corporation of Redmond, Washington, modified as appropriate. Such operating systems may be modified or created in accordance with the present disclosure. Embodiments of the present disclosure provide systems and methods for processing medical images.

[0070] FIG. 3 illustrates a flow chart of an embodiment of a method for identifying organ types in volumetric medical images.

[0071] In step 305, a volumetric medical image 305 (see FIG. 3A below) is received. The volumetric medical image 305 may be received within the data processing system 101, for example, via the network interface 104 or the input device 203.

[0072] The volumetric medical image 305 illustrated in Figure 3A is composed of a three-dimensional array of voxels (volume elements) 306, 307, and 308. The array is illustrated in Figure 3A as a cube viewed from a perspective. The cube includes rows and columns of voxels 306, 307, and 308 extending along all three dimensions x, y, and z. To facilitate understanding of Figure 3A, only a portion of the voxels 306, 307, and 308 within the cube are illustrated.

[0073] Instead of a three-dimensional array, the methods described herein may also use multiple slices (two-dimensional arrays of pixels or picture elements) that together describe a (three-dimensional) volume. Indeed, any other data structure that contains values, such as intensity, and describes three-dimensional space may also be used. Such values ​​are referred to herein as "voxels." The values ​​may be combined with information that describes three-dimensional relationships with respect to other values, or the three-dimensional relationships may be inferred from the data structure or any other source.

[0074] The volumetric medical image 305 includes at least one organ 309 or portion of an organ 309. In the example of Figure 3A, a portion of the organ 309, e.g., a lung or kidney, is represented by hashed (hatched) voxels 308 in the volumetric medical image 305.

[0075] In step 302, a single point of interest 310 in a volumetric medical image is received. As described in more detail below, the single point of interest 310 may be selected by a user via a graphical user interface using a pointing device such as a mouse. The single point of interest 310 may be received via, for example, the network interface 104 or the input device 204 (see FIG. 2).

[0076] A single point of interest 310 is a point in a volumetric medical image that is sought to identify the type of organ 309 that corresponds to that point. This single point of interest 310 can be described using x, y, and z coordinates and can correspond to a particular voxel within a cube. For example, if a mouse is used to select the single point of interest 310, for example, by clicking or pausing the cursor on the image, once the selection of the single point of interest 310 is made, the x, y, and z coordinates of the mouse cursor are sent to the data processing system 101, for example, via the input device 204.

[0077] In step 303, voxels 306, 308 are sampled from the volumetric medical image 305. For this purpose, for example, the module 103 (see FIG. 2) reads the voxels 306, 308 from the volumetric medical image 305 contained in the database 102. Between two sampled voxels 306, 308, at least one voxel 307 is skipped. In FIG. 3A, sampled voxels are represented using a check mark, while skipped or unsampled voxels are represented using a cross (x). In the embodiment of FIG. 3A, only every other voxel 306, 308 is sampled in a row of voxels defined by adjacent voxels 306, 307, 308. These voxels 306, 308 may be sampled consecutively or in parallel.

[0078] The module 103 may include a corresponding sampling model, which may be an algorithm or other type of code, to define which voxels are sampled and which are not. Alternatively, the sampling model may be defined by a user via the input device 204, or the sampling model may be provided via the network interface 104.

[0079] In an embodiment, in step 303, the voxels 306, 308 are sampled in a sparse manner. That is, the number of sampled voxels in the volumetric medical image 305 is less than the total number of voxels contained in the volumetric medical image. In particular, the number of sampled voxels is considered sparse if it is less than 50%, or less than 20%, or less than 10% of the total number of voxels. In one embodiment, the sampled voxels 306, 308 are less than 1%, preferably less than 0.1%, and more preferably less than 0.01% of the total number of voxels in the volumetric medical image 305.

[0080] It is noted that voxels may be sampled in a random manner (unless at least one voxel is skipped between two sampled voxels). For example, a random or pseudo-random number generator may be used to identify voxels 306, 307, 308 to be sampled in the volumetric medical image 305, as well as to identify other voxels not to be sampled. For example, the random or pseudo-random number generator may be part of or used to provide the sampling model.

[0081] In particular, the inventors have found that it is advantageous if, in step 303, the voxels 306, 308 are sampled at a sampling rate per unit length, area, or volume that decreases with distance 311 from a single point of interest 310. Furthermore, it has been found that improved results are obtained even more if the sampling rate decreases at a non-linear rate, particularly an exponential, logarithmic, or power function rate.

[0082] In this regard, reference will be made to FIGS. 4 and 5, which illustrate a sparse sampling model 400 (only portions of the sampling model 400 are labeled 400a, 400b, and 400c for ease of understanding) in a two-dimensional slice 401 acquired from a volumetric medical image 305. The sampling model 400 is configured to provide sparse sampling, where the sampling rate per unit volume decreases with distance along the x, y, and z directions. While the two-dimensional image is illustrated in FIG. 4, the y-axis direction, which corresponds to the depth direction perpendicular to the plane illustrated in FIG. 4, is not illustrated, but is illustrated corresponding to the sampling rates shown in FIG. 4 for the x and z directions.

[0083] FIG. 5 illustrates three-dimensional grids with different grid spacings that may be used to define the sampling model 400 illustrated in FIG. 4 . Cubes (or cubic bones) are represented by reference numerals 501, 502, and 503. Cubes 501, 502, and 503 are nested within one another, with the smallest cube 501 having a single interest point 310, e.g., at its center. Each cube includes a grid. Cube 501 includes grid 504, cube 502 includes grid 505, and cube 503 includes grid 506. These grids 504, 505, and 506 may be regularly spaced to define the cubes or cubic bones therein. For illustrative purposes, these grids 504, 505, and 506 are illustrated in only two dimensions (i.e., the plane of this paper) and are only partially shown. The grid spacing is defined as the distance between two adjacent nodes of each grid. Any node is defined as the intersection of three grid lines. As shown in detail for grid 506, two nodes 507 and 508 are separated by a distance D6. Similarly, the grid spacing of grid 504 is defined by a distance D4, and the grid spacing of grid 505 is defined by a distance D5.

[0084] In our experiments, D4 was chosen to be 8 mm, D5 to be 20 mm, and D6 to be 80 mm. Nodes 507 and 508 can be considered to be nodes within the volume of each cube (or cubic bone), excluding the volume of the largest cube (or cubic bone) nested within that cube. For example, for cube 503, nodes 507 and 508 can be considered to be inside the volume of cube 503 that is not located within the volume of cube 502.

[0085] Nodes 507 and 508 define the sampling model 400 and, therefore, the voxels 306 and 308 (see FIG. 3A) sampled in the volumetric medical image 305. In this case, when the volumetric medical image 305 was sampled using the sampling model 400, 1,195 voxels were sampled (i.e., intensity values ​​retrieved from, for example, the database 102). Meanwhile, the total number of voxels in the volumetric medical image was 25,000,000. Thus, less than 0.1% of the total number of voxels were sampled. FIG. 6 illustrates the voxels sampled within the slice of FIG. 4, along with the image.

[0086] Referring again to Figure 3, an illustrated step 304 identifies the type of organ 309 at a single point of interest 310 by applying a trained classifier to sampled voxels 306, 308. The trained classifier is a neural network, particularly a multilayer perceptron, a convolutional neural network, a Siamese network, or a triplet network.

[0087] 4 and 5 illustrate the effectiveness of the method devised by the present inventors.

[0088] First, for comparison purposes, segmentation (or parcellation) can be used to find labels or landmarks (organ types) for each voxel in the volumetric medical image 305 of Figure 4, which illustrates a two-dimensional slice. In particular, the U-Net segmentation algorithm can be used. U-Net is a convolutional neural network developed at the Department of Computer Science at the University of Freiburg for the segmentation of biomedical images. Segmentation typically takes 10 seconds and is described in the following paper: Hatamizadeh, Ali et al., "UNETR: Transformers for 3D Medical Image Segmentation," arXiv:2103.10504v3 (2021)

[0089] In contrast, our experiments have shown that it only took 10 ms to find the label (i.e., organ type) at a single interest point 310 using a trained classifier with descriptors (derived using the sampling model 400). The trained classifier used in this experiment is a Residual Neural Network (ResNet) classifier. ResNet utilizes a very deep feedforward neural network with hundreds of layers.

[0090] When we applied the ResNet classifier in a typical manner, i.e., sampling the total number of voxels in the volumetric medical image 305 to find the label (organ type) at a single interest point 410, the data processing took approximately 1 second.

[0091] FIG. 7 illustrates a portion of a graphical user interface 700 displayed on, for example, one of the client-side devices 107A-N in an embodiment. In FIG. 7, the graphical user interface 700 illustrates a slice of a volumetric medical image 305, with a mouse cursor 701 hovering over a portion of an organ 309. By hovering the cursor 701 over the volumetric medical image 305 for a predetermined period of time, a user can select a single point of interest 310 in the volumetric medical image 305. Accordingly, the corresponding coordinates of the single point of interest 310 are transmitted to, for example, module 103 (see FIG. 2), which can then perform steps 301-304 to determine which organ type corresponds to the single point of interest 310.

[0092] In one embodiment, organ 309 is found to be a lung, and module 103, or another piece of software and / or hardware, therefore generates semantic descriptive data (e.g., for "lung") and displays that data at or near a single point of interest 310 (see, e.g., FIG. 7).

[0093] In FIG. 8 , a user is using the graphical user interface 700 to perform a measurement 800 on an organ 309. For example, the organ 309 has been identified as a "left kidney" for a single point of interest 310 using method steps 301-304. Measurements 800 are performed, such as tumor size, organ 309 size, or the like. The module 103, or any other software and / or hardware, can be configured to store the measurement (or measurements) 800 (in this example, the length indicated by the 40 mm distance line) and a label (i.e., the type of organ found at the single point of interest 310 in association with the measurement 800) in a database 900, as illustrated in FIG. 9 . The database 900 can also be stored in the memory 203 (see FIG. 2 ). Within the database 900, multiple measurements 800 are stored along with semantic descriptive labels 702 and coordinates 901 (x, y, z) of the corresponding single point of interest 310. The information 800, 702, 901 may be stored in an array 902. In the database 900, multiple arrays 902 may be stored.

[0094] FIG. 10 illustrates an example of applying the method illustrated in FIG. 3 to federated learning. A database 102 stores a plurality of volumetric medical images 305. In one embodiment, module 103 uses a predetermined single interest point in each image 305 and a predetermined classifier 400 to identify the type of organ 309 depicted in each image 305. Depending on the identified organ type, module 103 provides images 305 found to represent lungs to a lung nodule classifier (classification device) 1001. Images 305′ found to represent livers are provided to a liver lesion classifier 1002. Images 305″ found to represent kidneys are provided to a kidney lesion classifier 1003.

[0095] In particular, a method of federated learning can be implemented as illustrated in FIG.

[0096] Initially, module 103 receives (step 1101) untrained classifiers 1001-1003, for example. Once trained, untrained classifiers 1001-1003 are configured to identify organ-specific lesions (abnormalities). For example, lung nodule classifier 1001 is trained to detect nodules within the lungs. Similarly, lesion classifier 1002 is trained to detect lesions in the liver, etc.

[0097] In step 1102, the module 103 selects from the untrained classifiers 1001-1003 that correspond to the particular organs in the volumetric medical image 305 identified in step 304 (see FIG. 3).

[0098] In step 1103, the untrained classifiers 1001-1003 are trained using volumetric medical images 305 showing the corresponding organs.

[0099] For training purposes, organ-specific anomalies need to be known for each volumetric medical image 305. Such information is preferably also stored in the database 102, which is used during the training step 1102.

[0100] Advantageously, in the method illustrated in FIGS. 10 and 11, the classifier in step 304 can be used as a gating mechanism to direct input data (particularly the volumetric medical image 305) to accurate organ-specific classifiers 1001-1003. Therefore, complete segmentation is not required to effectively train the classifiers 1001-1003. This gating mechanism can be used to train the classifiers 1001-1003, but can also be used in the application stage. Furthermore, the method of FIG. 3 can be applied to multiple single points of interest 310 in the same volumetric medical image 305. If multiple organs are identified in the same volumetric medical image 305, the volumetric medical image may be segmented or divided into different sub-images, each representing one organ 309. The sub-images may then be used to train the organ-specific classifiers 1001-1003. If the organ identified in the imagelet corresponds to that of an organ-specific classifier 1001-1003, then that classifier 1001-1003 is trained on that image.

[0101] In the application phase, a trained organ-specific classifier 1001-1003 is selected, for example by module 103, based on the organ identified in the processing steps following step 304, and the trained organ-specific classifier 1001-1003 is then applied to the volumetric medical image 305 or a sub-image thereof. Thus, the method is not only suitable for identifying organ types, but also for identifying specific abnormalities within organs.

[0102] In the embodiment illustrated in FIG. 12 , steps 301-304 (see FIG. 3 ) are performed for N single interest points in the volumetric medical image 305. For example, a coarse grid is selected, such as grid 506 in FIG. 5 (grids 504 and 505 do not exist in this example, but grid 506 fills the entire volume of cube 503). Each node 507, 508 is selected as a respective single interest point 310. Next, voxels 306, 308 are sampled, for example, using the same sampling model 400 (although, if possible, the sampling model may be moved so that the single interest point 310 remains at the center of the sampling model 400 when applied to the volumetric medical image 305). As a result, in each step 304, the type of organ 309 at each interest point 507, 508 is identified. Steps 301-304 may be performed in parallel or sequentially for each node 507, 508. The grid spacing, i.e., distance D6 (see FIG. 5), is selected to be larger than the spacing between two adjacent voxels 305, 306, and 307 in the volumetric medical image 305. This results in a coarse-level segmentation, as shown in FIG. 12. Still, no segmentation neural network is used, but a classifier is applied to obtain this coarse-level segmentation. In the example of FIG. 12, distance D6 (see FIG. 5) is equal to 1 cm (resolution). To obtain the segmentation illustrated in FIG. 12, the CPU hardware required only 0.41 seconds without any extra preprocessing steps.

[0103] FIG. 13 illustrates an embodiment of a method for training the classifier used in step 304 (see FIG. 4).

[0104] In step 1201, a volumetric medical image 305 is received, along with an identifier (label) identifying the type of at least one organ 309 depicted in the volumetric medical image 305. This identifier may correspond to the semantic descriptive label 702 referenced in Figures 7 and 8, or may be in any other suitable form. This identifier may be obtained by applying a full segmentation to the volumetric medical image 305 prior to step 1201, thus providing an identifier for every voxel in the image 305.

[0105] In step 1202, a single interest point 310 is received in a volumetric medical image 305. For training purposes, for example, the single interest point may be predetermined or randomly generated.

[0106] In step 1203, voxels are sampled from the volumetric medical image 305, with at least one voxel 307 skipped between two sampled voxels 306, 308.

[0107] In step 1204, an untrained classifier is applied to the sampled voxels 306, 308 to identify the type of organ 309 at the single point of interest 310.

[0108] The features, advantages, and explanations discussed with respect to the method of Figure 3 are equally applicable to steps 1201-1204. In particular, the sampling in step 1203 can be performed using sampling model 400 as described above, thereby speeding up the training process and increasing its reliability.

[0109] In step 1205, the identified type of organ 309 is compared to the received organ type.

[0110] In step 1206, the classifier is modified based on the comparison, e.g., weighted to provide a trained classifier. Steps 1201-1206 may be performed repeatedly for different volumetric medical images 305 until the classifier is fully trained. Steps 1201-1206 may be performed using the device 101 or system 100 illustrated in Figures 1 and 2. In particular, module 103 or any other hardware and / or software may be used to perform steps 1201-1206.

[0111] The above examples are presented for illustrative purposes only and should not be construed as limiting the invention disclosed herein. While the present invention has been described with reference to various embodiments, the terminology used herein is for purposes of description and illustration only and should not be construed as limiting. Furthermore, while the present invention has been described with reference to particular means, materials, and embodiments, the present invention is not limited to those specifically disclosed herein; rather, the present invention extends to all functionally equivalent or similar structures, methods, uses, and the like, within the scope of the appended claims. Based on the teachings herein, those skilled in the art will be able to make various modifications and variations in its aspects without departing from the scope and spirit of the present invention. [Explanation of symbols]

[0112] 100: System 101: Computer-Implemented Apparatus (or Device) 102: Medical database 103: Module 104: Network Interface 105: Network 107A-107N: Client-side device (or user-side device) 108: Medical imaging unit (or medical imaging device) 201: Processing device 202: Memory 203: Storage device 204: Input device 205: Bus 206: Output device 301-304: Each step of the method 305: Volumetric Medical Imaging 305': Volumetric Medical Imaging 305): Volumetric medical imaging 306-308: Voxel (or volume element) 309: Organ (or organs) 310: Single point of interest 311: Distance x, y, z: orthogonal directions in space 400: Sampling model 400a-400c: point 401: Slice 501-503: Cube 504-506: Lattice (or Grid) 507, 508: Nodes (or nodes) D4, D5, D6: Distance 700:HMI 701: Cursor 702: Label (or sign) 800: Measurement line 900: Database 901: Coordinates 902: Array 1001-1003:Classifier 1101-1103: Method steps 1201-1206: Method steps

Claims

1. 1. A computer-implemented method for identifying organ (309) types in volumetric medical images (305), comprising: a) receiving a volumetric medical image (305) including at least one organ (309) or part of an organ; b) receiving a single point of interest (310) in the volumetric medical image (305); c) sampling voxels (306, 308) from said volumetric medical image (305), skipping at least one voxel (307) between two sampled voxels (306, 308); d) applying a trained classifier to the sampled voxels (306, 308) to identify the organ type at the single point of interest (310); Each step of the sampled voxels (306, 308) are less than 1% of the total number of voxels (306, 307, 308) in the volumetric medical image (305); method.

2. The method of claim 1 , wherein step c) samples the voxels (306, 308) in a sparse and / or random manner.

3. 2. The method of claim 1, wherein step c) samples the voxels (306, 308) at a sampling rate per unit length, area, or volume that decreases with the distance (311) from the single point of interest (310) to each voxel (306, 308).

4. The method of claim 3 , wherein the sampling rate decreases at a non-linear rate, including an exponential, logarithmic, or power-function rate.

5. The method of claim 1 , wherein the sampled voxels (306, 308) are less than 0.1% of the total number of voxels (306, 307, 308) in the volumetric medical image (305).

6. The method of claim 1 , wherein the trained classifier is a neural network, including a multi-layer perceptron, a convolutional neural network, a Siamese network, or a triplet network.

7. 2. The method of claim 1, wherein the volumetric medical image (305) or a portion thereof including the single point of interest (310) is displayed on a graphical user interface (700), and wherein a semantic description (702) of the type of identified organ (309) is generated and displayed at or near the location of the single point of interest (310).

8. The single point of interest (310) is selected by a user, and / or the single point of interest (310) is selected by pausing a cursor (701) manipulated by a user over the volumetric medical image (305) or a portion thereof displayed on a graphical user interface (700); and / or When a user performs a measurement (800) on the volumetric medical image (305) or a portion thereof, the identified type of organ (309) is stored in a database (900) together with the measurement (800). The method of claim 1.

9. receiving a plurality of untrained classifiers (1001-1003) for identifying organ-specific anomalies; selecting one or more untrained classifiers (1001-1003) from the plurality of classifiers based on the identified organ (309); training the one or more untrained classifiers (1001-1003) using the volumetric medical images (305); The method of claim 1 , comprising:

10. performing and / or repeating steps a) to d) for each of "N-1" single points of interest (310) in the volumetric medical image (305), where "N" is less than or equal to the total number of voxels (306, 307, 308) in the volumetric medical image (305); Executing steps a) to d) in parallel in one or more processing devices (201), and / or The "N" is a grid spacing (D) that exceeds the spacing of the voxels (306, 307, 308) contained in the volumetric medical image (305). ６ ) corresponds to a lattice (506) with The method of claim 1.

11. An apparatus (101) for identifying a type of organ (309) in a volumetric medical image (305), comprising: one or more processing devices (201); a first receiving device (104, 204) configured to receive one or more volumetric medical images (305) captured by a medical imaging device (108); a second receiving device (104, 204) configured to receive a single point of interest (310) in the one or more volumetric medical images (305); a memory (202) coupled to said one or more processing devices (201) and including modules (103) configured to perform the steps of the method according to any one of claims 1 to 10; An apparatus comprising:

12. A system (100) for identifying organ (309) types in volumetric medical images (305), comprising: One or more servers (101); a medical imaging device (108) coupled to said one or more servers (101); Including, The one or more servers (101) include instructions that, when executed, cause the one or more servers (101) to perform each step of the method of any one of claims 1 to 10.

13. A computer program product comprising machine-readable instructions that, when executed by one or more processing devices (201), cause the one or more processing devices (201) to perform the steps of the method of any one of claims 1 to 10.

14. A computer-readable medium having stored thereon program code portions of a computer program, the program code portions being loadable and / or executable in a system (100) such that, when executed in the system (100), the program code portions cause the system (100) to perform the steps of the method of any one of claims 1 to 10.

Images