Measurement of human body tissue

Abstract

To provide an improved method for measuring human body tissue from a set of medical images representing the human body tissue.SOLUTION: A method applies a trained neural network to a set at S10. Thus, the method identifies one or more segments of the human body tissue for at least two images of the set at S20. The method also computes a bounding box enclosing each segment at S30. The method also determines an intersection between a pair of bounding boxes at S410. The method determines an intersection between the segments when the pair of intersection is non-empty S420. The method merges the segments by computing a resulting bounding box enclosing the segments when the intersection between the segments is non-empty at S430. The method also measures the size of the segment provided for the resulting bounding box at S50.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to the field of computer programs and systems, and more particularly, to methods, systems, and programs for measuring human tissue from a set of medical images.

Background Art

[0002] Medical imaging is becoming increasingly important in clinical trials. In oncology, 95% of research uses medical imaging to evaluate the effectiveness of treatments measured by the time to progression of tumor disease (e.g., progression-free survival) or objective response rate (ORR).

[0003] There are numerous systems and programs for analyzing medical images such as CT scans. For example, 3DS MEDIDATA RAVE Imaging provides an IT infrastructure for imaging data collection for intensive image review based on the RECIST (Response Evaluation Criteria In Solid Tumors) criteria, which are international best practice recommendations. The document "Eisenhauer, E.A., et.al, New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). European journal of cancer, 45(2), 228-247, (2009)" is a revised guideline of the RECIST criteria that defines the standard approach for solid tumor measurement and the definition of objective assessment of tumor size changes for use in clinical trials of adult and pediatric cancers. 3DS MEDIDATA RAVE Imaging relies on the expertise of radiologists. The process of measuring solid tumors is manual, repetitive, cumbersome, and involves measurement errors and variability among experts. Figure 1 shows an example of annotations 1010 - 1040 of lesions identified using the RECIST process and 50% consensus 1050. There are multiple differences in annotations 1010 - 1040 due to variability among radiologists responsible for manually correcting medical images to identify and measure target lesions in the human body and to identify new lesions. In practice, a radiologist selects one image from the set using their best practice and performs tissue measurements on that image.

[0004] An approach has been developed to predict 2D segmentation of tumors on CT scans. One of these approaches is described in the paper Yan, K., et al, MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation (MICCAI2019), which relies on a maskrcnn model specialized for processing on CT scans to predict 2D segmentation of tumors. All CT scan slices are annotated with all tumors detected therein. However, those measurements are not very useful for radiologists, because to comply with the RECIST standard, annotations have to be made on a single slice for each lesion. In other words, the intervention of radiologists is still required to interpret CT scans. Furthermore, in this approach, for performance reasons, one special model is required for each organ.

[0005] The related paper Cai, J., Harrison, A. P., Zheng, Y., Yan, K., Huo, Y., Xiao, J.,… & Lu, L. (2020). Lesion-harvester: Iteratively mining unlabeled lesions and hard-negative examples at scale. IEEE Transactions on Medical Imaging, 40(1), 59-70 describes a method for detecting the 3D bounding boxes of related tumors, but it does not provide automatic segmentation nor RECIST measurements. Here too, the intervention of radiologists is required to interpret CT scans.

[0006] Therefore, the main problems of the above methods are that the accuracy of CT scan interpretation depends on the expertise of radiologists, and the process is manual, iterative, and cumbersome.

Summary of the Invention

Problems to be Solved by the Invention

[0007] In such a situation, there is still a need for an improved method for measuring human tissue from a set of medical images representing the human tissue.

Means for Solving the Problem

[0008] Accordingly, a computer-implemented method for measuring human tissue from a set of medical images representing the human tissue is provided. The method comprises obtaining a trained neural network configured to output segments of human tissue from a set of medical images. The method also comprises applying the trained neural network to the set of medical images. Thereby, the method identifies one or more segments of human tissue for at least two of the images in the set. The method also comprises calculating a bounding box that encloses each segment. The method also comprises determining an interaction between pairs of the bounding boxes. The method also comprises, if the pair interaction is non-empty, determining an intersection between the segments enclosed by the pair of bounding boxes. The method also comprises, if the intersection between the segments is non-empty, merging the segments by calculating a resulting bounding box that encloses the segments. The method also comprises measuring the size of the segments comprised in the resulting bounding box.

[0009] The method may comprise one or more of the following. - The method, before determining the intersection between segments, calculates, for each identified segment, a 2D bounding box from at least two of the images in the set, each 2D bounding box enclosing the identified segment, and determines an intersection between at least two of the calculated 2D bounding boxes and further comprises, wherein the determination of the intersection between segments is only performed for segments for which the intersection between the 2D bounding boxes is non-empty. - The method further comprises iteratively determining intersections across pairs of bounding boxes by repeating the determination and merging for each remaining pair of bounding boxes. - The method further comprises maintaining a pair of bounding boxes surrounding each segment when the intersection between segments is empty. - The method further comprises obtaining the distance between images comprising segments surrounded by a pair, and the determination of the intersection between segments surrounded by a pair is performed only when the distance between images comprising segments is below a predetermined threshold. - The trained neural network is further configured to output tags identifying segments of human tissue, and for segments having the same tag, the determination of intersections across pairs of bounding boxes is performed. - The determination of intersections between segments comprises obtaining a mask for each segment, calculating the intersection between the obtained masks, and segment merging is performed only for segments for which the intersection between the obtained masks is not empty. - The trained neural network is further configured to output 2D bounding boxes surrounding segments of human tissue, and the calculation of the 2D bounding boxes is performed by the trained neural network. - Measuring the size of a segment further comprises selecting the segment having the longest diameter among those provided in the resulting bounding box. - The set of medical images comprises a set of CT-SCAN images, a set of MRI images, a set of PET scan images, or a set of ultrasound images. - The human tissue represented by the set of medical images corresponds to a lesion of an organ, or an aneurysm or tissue of an organ.

[0010] Also provided is a method for identifying the progression of human tissue. This method for identifying the progression of human tissue comprises obtaining a current set of medical images representing a patient's human tissue. This method for identifying the progression of human tissue also comprises using a method for obtaining a current measurement of the size of a segment. This method for identifying the progression of human tissue also comprises obtaining measurements obtained from a past set of medical images representing a patient's human tissue. This method for identifying the progression of human tissue also comprises calculating the difference between the current measurement and the past measurement. This method for identifying the progression of human tissue also comprises thereby identifying the progression of human tissue.

[0011] Furthermore, when a program is executed by a computer, there is provided a computer program comprising instructions for causing the computer to implement the methods disclosed herein.

[0012] Furthermore, there is provided a computer-readable storage medium having a computer program recorded thereon.

[0013] Furthermore, there is provided a system comprising a processor coupled to a memory having a computer program recorded thereon. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Next, non-limiting examples will be described with reference to the accompanying drawings.

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0016] Referring to the flowchart of FIG. 2, a computer-implemented method for measuring human tissue from a set of medical images representing human tissue is proposed. This method (also referred to as the "measurement method") comprises obtaining a trained neural network configured to output segments of human tissue from the set of images (S10). The method also comprises applying the trained neural network to the set of medical images (S20). The method thereby identifies one or more segments of human tissue for at least two of the images in the set. The method also calculates a bounding box surrounding each identified segment (S30). The method also determines an intersection between pairs of bounding boxes (S410). If the intersection of the pair is not empty, the method determines an intersection between the segments enclosed by the pair of bounding boxes (S420). If the intersection between the segments is not empty, the method merges the segments by calculating a bounding box surrounding the segments (S430). The method also measures the size of the segments comprised in the resulting bounding box (S50).

[0017] By such a method, the measurement of human tissue on a set of medical images representing human tissue is improved.

[0018] In particular, in this method, an accurate measurement of human tissue can be obtained by the method of segment determination. In fact, since the resulting bounding box is provided with segments, this method ensures that the segments provided in the resulting bounding box belong to the same human tissue, thus accurately determining an anatomically correct human tissue representation. For example, a human tissue can appear as a single segment in one image of a set of images, while multiple segments of that tissue can be shown in another image. Structurally, the resulting bounding box includes segments of two images in the set that represent the same tissue. Therefore, this method eliminates user bias when determining the appropriate segments for performing measurements of human tissue.

[0019] As described above, this method can accurately determine an anatomically correct human tissue representation by calculating the resulting bounding box. This is all due to the specific steps performed by this method. First, this method utilizes the neural network paradigm to identify one or more segments of human tissue for at least two images in the set. This method calculates the bounding box surrounding each segment and determines the intersection between pairs of bounding boxes. Therefore, this method performs a rough detection of segments (belonging to pairs of bounding boxes) that are likely to belong to the same human tissue. Such a rough detection is computationally efficient and particularly fast. If the intersection of a pair of bounding boxes is not empty, this method determines the intersection between the segments enclosed by the pair of bounding boxes. In other words, this method performs a fine-grained determination of segments that belong to the same human tissue. This fine-grained determination enables this method to avoid false detections (e.g., bounding boxes that intersect but have no intersecting segments) that may be retained by the rough detection. The segments merged by calculating the resulting bounding box thus include segments that belong to the same human tissue.

[0020] Furthermore, a method for identifying the progression of human tissue (also referred to as the "identification method") is provided. The identification method comprises obtaining a set of current medical images representing a patient's human tissue. Obtaining the set of current medical images may comprise obtaining such a set from non-volatile storage or obtaining the set from a network.

[0021] The identification method also comprises using a measurement method to obtain a current measurement of the size of the segment. In other words, the measurement method receives the set of current medical images as input and outputs a measurement of the size of the segment comprised in the resulting bounding box. The set of current medical images is obtained at any time.

[0022] The identification method also comprises obtaining measurements obtained from a set of past medical images representing a patient's human tissue. In other words, the measurement method receives the set of past medical images as input and outputs a measurement of the size of the segment comprised in the resulting bounding box. The set of past medical images represents the same human tissue as the current measurements. The set of past medical images was taken at a time prior to the time at which the set of current medical images was obtained (thus, the times at which the past set and the current set were obtained may differ by, for example, several hours, several days, or even several months).

[0023] The identification method also comprises calculating the difference between the current measurement and the past measurement. The difference may be a simple subtraction. The identification method thereby identifies the progression of the human tissue. In fact, the identification indicates the progression of the human tissue according to the change in the dimensions of the segment (represented by the difference) for which the measurements are determined.

[0024] The methods disclosed in this specification are implemented by a computer. This means that the step(s) (or substantially all steps) of the method(s) are performed by at least one computer, or any similar system. Thus, the step(s) of the method(s) are performed by a computer, perhaps completely automatically, or semi-automatically. In an example, the trigger for at least some of the step(s) of the method(s) can be performed by an interaction between a user and the computer. The required level of user-computer interaction can vary depending on the expected level of automation and is balanced with the need to fulfill the user's wishes. In an example, this level can be defined and / or pre-defined by the user.

[0025] A typical example of the computer implementation of a method is to execute the method on a system adapted for this purpose. The system includes a processor coupled to a memory and may include a graphical user interface (GUI). The memory stores a computer program comprising instructions for executing the present method. The memory can also store a database. The memory is any hardware suitable for such storage and perhaps comprises a plurality of physically different parts (e.g., for the program and perhaps for the database).

[0026] Figure 3 shows an example of a system, which is a client computer system, for example, the user's workstation.

[0027] The client computer in this example includes a central processing unit (CPU) 3010 connected to an internal communication BUS 3000, and a random access memory (RAM) 3070 also connected to the BUS. The client computer further includes a graphics processing unit (GPU) 3110 associated with a video random access memory 3100 connected to the BUS. The video RAM 3100 is also known as a frame buffer in the art. A mass storage controller 3020 manages access to a mass storage device such as a hard drive 3030. Mass storage devices suitable for tangibly embodying computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and any form of non-volatile memory. Any of the above may be supplemented by, or incorporated in, a specially designed application specific integrated circuit (ASIC). A network adapter 3050 manages access to a network 3060. The client computer may also include tactile devices 3090 such as a cursor control device, a keyboard, etc. The cursor control device is used in the client computer to enable the user to selectively place the cursor at any desired location on a display 3080. Further, the cursor control device enables the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals into the system. Typically, the cursor control device may be a mouse, and the buttons of the mouse are used for signal generation. Alternatively, or in addition, the client computer system may include a sensing pad and / or a sensing screen.

[0028] A computer program may comprise instructions executable by a computer, the instructions comprising means for causing the system to execute the method. The program may be recordable on any data storage medium including the memory of the system. The program may be implemented, for example, in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. The program may be implemented as an apparatus, such as a product tangibly embodied in a machine-readable storage device for execution by, for example, a programmable processor. The steps of the method may be executed by a programmable processor executing a program of instructions that operate on input data to produce output, thereby performing the functions of the method. Accordingly, the processor may be programmable and coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device and to transmit data and instructions to them. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as required. In any case, the language may be a compiled or interpreted language. The program may be a full-installation program or an update program. Applying the program to the system in any case results in instructions for executing the method. Alternatively, the computer program may be stored on and executed from a server in a cloud computing environment, the server communicating with one or more clients via a network. In such a case, the processing unit executes the instructions comprised by the program, thereby executing the method in the cloud computing environment.

[0029] A set of medical images may be formed by a data structure comprising a plurality (e.g., two or more) of medical images (by storing them in a physical memory such as, for example, a hard drive).

[0030] In an example, the set of medical images can comprise a set of CT scan images, or a set of MRI images, or a set of PET scan images, or a set of ultrasound images. A CT scan (Computed Tomography scan) is a specific type of medical image created by irradiating the human body with X-rays. Signals are read and analyzed to reconstruct high-density volumes of the body. In practice, the volume is divided into slices whose normal direction is from the feet to the head. An MRI (Magnetic Resonance Imaging) is a specific type of medical imaging technique that uses a magnetic field and computer-generated electromagnetic waves to create detailed images of organs and tissues within the body. A PET scan (Positron Emission Tomography) is an imaging test that can help reveal the metabolic or biochemical functions of tissues and organs. In a PET scan, a radioactive drug called a tracer is used to show both typical and atypical metabolic activities. In ultrasound imaging (ultrasound examination), high-frequency sound waves are used, and an image is generated based on the reflection of sound waves from body structures. Information necessary to generate the image is provided from the intensity (amplitude) of the sound signal and the time it takes for the sound wave to pass through the body.

[0031] The set of medical images represents human tissue. Tissue, as known in the art, is a group of adjacent cells organized to perform one or more specific functions. Thus, human tissue can correspond to a lesion in an organ (such as the liver, heart, or bone), or to tissue corresponding to an aneurysm, or can be an organ (e.g., a part of an organ or the whole organ). A lesion can be an abnormality in the human body that does not presuppose pathogenicity of human tissue. As an example, in the context of oncology, the term lesion can be used to refer to a tumor.

[0032] A set of medical images can be images ordered according to a reference axis. In other words, each image can comprise a representation of human tissue as seen from a view at the position of the reference axis (for example, a view orthogonal to the reference axis). The reference axis can be set, by convention, for example, to a standard z-axis along a body part comprising human tissue, in which case each view can comprise a view on the x-y plane (that is, an axial view or a cross-sectional view of the human tissue). The set of images can comprise information on the position of the images with respect to the reference axis, that is, the position of the reference axis from which the representation of the human tissue is seen. The set of images can be ordered according to such information. In an example, the set of images can be evenly distributed, that is, the images sample the body part evenly along the reference axis (with the separation between two consecutive images in the ordered set being the same). For example, the images can be spaced 1 mm or more apart along the reference axis.

[0033] The method obtains a trained neural network (S10). This means that a computerized system executing the method can directly access the neural network (for example, the neural network is executed on the computerized system or is executed by the computerized system), or can indirectly access the neural network (for example, the computerized system can provide an input to the neural network and obtain the output of the neural network for the input). The trained neural network is configured to output segments of human tissue from a set of images. In other words, the trained neural network performs segmentation of objects within the images. Image segmentation (also referred to as object segmentation) groups pixels belonging to the same object by classifying each pixel value of the image into a specific class, as is known in the art.

[0034] A neural network can be a function that comprises a collection of connected nodes (also called "neurons"). Each neuron receives an input and outputs a result to other neurons it is connected to. The neurons and the connections that link the neurons respectively have weight values that are adjusted by training. In the context of the present disclosure, "trained neural network" means that the weights of the neural network are deterministically saved after learning / training is performed on a dataset. The dataset can comprise a set of medical images each comprising one or more annotated segments of human tissue. The dataset can comprise medical images from the DeepLession dataset, for example, as described in the document Yan K, Wang X, Lu L, Summers RM, DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning, J Med Imaging (Bellingham), 2018.

[0035] The trained neural network can be a deep convolutional neural network such as the Mask R-CNN neural network described in the document Kaiming He, Georgia Gkioxari, Piotr Dollar, Ross Girshick Mask R-CNN, arXiv:1703.06870.

[0036] This method applies a trained neural network to a set of medical images (S20). In other words, the set of medical images is provided as the input to the trained neural network. Thereby, the trained neural network identifies one or more segments of human tissue for at least two of the images in the set. One or more identified segments of human tissue are output by the trained neural network. In other words, this method processes a set of medical images as the input to the trained neural network and outputs one or more segments of human tissue according to the weight values of the trained neural network. As is known in the field of machine learning, processing the input by a neural network includes applying operations defined by data including weight values to the input.

[0037] A segment can be any collection of pixels corresponding to a region of the corresponding medical image (i.e., the medical image to which the trained neural network is applied). The pixels of the segment can have a color indicating the presence of human tissue. Thus, the segment may be called a 2D segment because the trained neural network performs 2D segmentation.

[0038] This method calculates a bounding box surrounding each segment identified by a trained neural network (S30). A "bounding box" means any geometric shape in two or more dimensions, for example, any three-dimensional geometric shape, such as a two-dimensional rectangle, a three-dimensional cuboid, etc. In the case of three dimensions, the bounding box may sometimes be called a bounding volume. The bounding box surrounds each segment. In other words, the bounding box comprises (or encloses) the pixels of the segment indicating the presence of human tissue. This method may take into account the spatial position of the segment. For example, this method may convert the segment into a 3D segment, and the 3D position of the segment may be the position relative to the z-axis defined by the x-y plane and the reference axis of its respective medical image. This method may determine the size of the bounding box from the difference between the positions of a pair of images along the reference axis, or from the difference between pairs of two consecutive images if the pairs of images are evenly distributed.

[0039] This method determines the intersection between pairs of bounding boxes (S410). Determining the intersection may include calculating the intersection of at least one point (or line or surface) belonging to each of the bounding boxes of the pair. This method may hold the positions of each bounding box according to the position of the segment in each image and with respect to the reference axis to determine the intersection.

[0040] The intersection of a pair of bounding boxes may be empty or non-empty. The intersection is empty if there is no geometric intersection or intersection / overlap between two elements (e.g., points, lines, or surfaces) corresponding to each of the bounding boxes of the pair. The intersection is non-empty if there is a geometric intersection or intersection / overlap between two elements of each of the bounding boxes of the pair.

[0041] If the intersection (S410) of the pair of bounding boxes is not empty, the method determines the intersection between segments enclosed by the pair of bounding boxes (S420). Determining the intersection between segments can include determining the intersection of at least one pair of points (or lines or surfaces) belonging to each segment enclosed by the pair of bounding boxes.

[0042] If the intersection between segments (S420) is not empty, the method merges the segments by calculating the resulting bounding box that encloses the segments (S430). In other words, the method creates a new bounding box that encloses the segments. When the method creates the resulting bounding box, it may discard the pair of bounding boxes that enclose the segments.

[0043] In an example, the method may further comprise maintaining the pair of bounding boxes that enclose each segment if the intersection between segments is empty. In other words, if the intersection is empty, the segments are not merged. This ensures that the segments are not unnecessarily merged, for example, when the segments do not correspond to the same human tissue or when the segments are too far apart.

[0044] The method measures the size of the segments comprised in the resulting bounding box (S50). The method may measure the diameter or cross-section of each segment.

[0045] For example, measuring the size of a segment may further comprise selecting the segment having the longest diameter (i.e., the longest distance) comprised in the resulting bounding box. The method may determine the longest diameter by calculating all the distances between the lines connecting pairs of points on the contour of the segment and holding the line with the maximum distance as the longest diameter. The method may also determine the minor diameter. The minor diameter (or short distance) may be determined from a line connecting two points on the contour that is orthogonal to the line corresponding to the longest diameter. The minor diameter may correspond to the line having the longest distance among the lines orthogonal to the line corresponding to the longest diameter.

[0046] This measurement can be performed based on such a longest diameter. For example, the measurement (S50) can be composed of the length of such a longest diameter. This is the case where the segment represents a human tissue such as a lesion other than a lymph node.

[0047] Alternatively, when the segment represents a human tissue such as a lymph node, the measurement is composed of the maximum segment of the segment orthogonal to such a longest diameter, that is, the short diameter.

[0048] Therefore, different types of measurement values can be provided, and the selection of the type of measurement value can vary depending on the type of human tissue represented in the image, for example, the RECIST diameter of a tumor lesion, the measured value of the diameter of an aneurysm, or the size of an organ. Therefore, the measurement of human tissue means obtaining at least one distance between two points on the representation of the human tissue. This distance can be a Euclidean distance.

[0049] In this method, since a trained neural network is applied to a set of medical images, one or more segments of human tissue are obtained particularly quickly and accurately.

[0050] A set of medical images representing human tissue can be regarded as a sampling of human tissue along a reference axis. By calculating the bounding box, it is possible to determine which segments belong to the same human tissue based on the spatial position of the bounding box. In other words, the bounding box provides a three-dimensional sense to one or more segments, thereby enabling the determination of which segments belong to the same human tissue according to the distribution of the images along the reference axis.

[0051] To identify which segments belong to the same human tissue, the method determines the intersection between pairs of bounding boxes. Thus, the method detects as early as possible cases where segments do not intersect. The intersection between pairs of bounding boxes can be regarded as a rough approximation. The method proceeds to a finer calculation. That is, after determining the intersection between pairs of bounding boxes, if such an intersection is determined to be non-empty, the method determines the intersection between pairs of segments and discards segments that are wrongly associated.

[0052] The method merges two segments by calculating the resulting bounding box and refines the determination of the human tissue for that measurement. Since the resulting bounding box encloses the intersecting segments, the method can measure the size of the segments belonging to the same human tissue. Thereby, the accuracy of the measurement is actually improved.

[0053] The method may further comprise calculating the 2D bounding box of each identified segment from at least two images of the set before determining the intersection between segments. Each 2D bounding box encloses the identified segment. The 2D bounding box can be a 2D parallelepiped (e.g., a square, a rectangle) having the minimum area to enclose the identified segment. The method may hold the 2D position of the segment (e.g., a position such as the center of the 2D bounding box) to place the corresponding 2D bounding box.

[0054] A trained neural network can also be configured to output a 2D bounding box surrounding a segment of human tissue. The calculation of the 2D bounding box can be performed by the trained neural network. In other words, the trained neural network can be applied to a set of medical images. Thereby, the trained neural network outputs the respective 2D bounding boxes of each segment. For example, the trained neural network can be a deep convolutional neural network such as a Mask R-CNN neural network that outputs a segment and the bounding box of each segment.

[0055] This method can also include determining an intersection between at least two of the calculated 2D bounding boxes. In other words, this method determines the intersection in the 2D coordinates of the calculated 2D bounding boxes. For example, determining the intersection between 2D bounding boxes can include determining the intersection of at least one pair of points (or lines) belonging to each 2D bounding box.

[0056] The determination of the intersection between segments can be performed only for segments where the intersection between the 2D bounding boxes is not empty. That is, this method does not require the determination of the intersection between segments when the intersection between the 2D bounding boxes is empty. Thereby, it is possible to perform a more accurate intersection detection than using 3D bounding boxes while being relatively fast compared to the detection and comparison of the intersection between segments.

[0057] The method may further comprise repeatedly determining (S40) intersections across pairs of bounding boxes by repeatedly performing determination (S410, S420) and merging (S430) for each remaining pair of bounding boxes. That is, following the determination of intersections (S410), the method may repeatedly determine the intersections between segments enclosed by pairs of segments (S420), and further, following (when the intersection between segments is not empty), merge the segments (S430). Since the method calculates the bounding box of each segment, the method may proceed to perform determination (S410) between other pairs of bounding boxes in another iteration after merging the segments (S430), or after determining that the intersection of a pair of bounding boxes is empty, or after determining that the intersection between segments is empty.

[0058] After the iterative determination (S40) is completed, the method may perform measurement (S50), for example, when it is determined that there are no more pairs of intersecting bounding boxes having non-empty intersections.

[0059] The method may further comprise obtaining the distance between images comprising segments enclosed by pairs. The distance between images may be obtained from the order of the images according to a reference axis. In other words, the distance may correspond to the difference in the position of each image with respect to the reference axis. The determination of the intersection between segments enclosed by pairs (S420) may be performed only when the distance between the images comprising the segments is below a predetermined threshold.

[0060] The trained neural network may also be configured to output a tag that identifies segments of human tissue. The tag may be any information indicating human tissue. The determination of intersections across pairs of bounding boxes is performed for segments having the same tag. In other words, the determination is performed for segments representing the same body tissue when captured by a set of images. This ensures that the measurements of human tissue are anatomically accurate.

[0061] The determination of the intersection between segments may include obtaining a mask for each segment. The mask may be a binary image consisting of zero values and non-zero values. The pixels corresponding to a segment may have a predetermined value, for example 1, and other pixels (for example, pixels corresponding to other parts of the human body) may have a value of zero. The determination of the intersection may also include calculating the intersection between the obtained masks. The calculation of the intersection may include determining the intersection at the pixel level. For example, if at least one pixel at a given position (the x-y position of the mask) of a given mask has the same value as the pixel at the same position of the given mask, it may be determined that the two binary masks intersect. The merging of segments may be performed only for segments where the intersection between the obtained masks is not empty. In other words, if the intersection is empty, the bounding box may be maintained. This makes the determination of the intersection finer and thereby more accurate.

[0062] Next, an example will be described with reference to FIGS. 4 to 9.

[0063] FIG. 4 shows an example of a pipeline for implementing the present method.

[0064] The pipeline 4100 may include obtaining a set of medical images by reading a DICOM CT scan image or a NIFTI image 4110 taken from a patient (for example, during an examination). DICOM refers to a dedicated file format for storing medical images. The DICOM format may include several modalities (for example, CT, RMI, US), and some of such modalities may include a text report or a description of a geometric object. The present method may be exemplified by steps 4120 to 4140, where step 4120 corresponds to applying a trained neural network to the set of medical images (S20), and steps 4130 and 4140 refer to steps S30 to S50 of the method. The result of the measurement may be output in the DICOM format (4150).

[0065] Interestingly, steps S10 to S50 of the present method can be implemented by a computer system interfaced between a device that generates images (e.g., a CT scanner) and a device that receives the images as input and displays them to a radiologist (e.g., a viewer). There is no need to adapt the two devices. In other words, the computer system that implements steps S10 to S50 of the method can be considered as a software plugin connected to the output of the image generation device and the input of the display device.

[0066] The set 4200 of medical images in the example comprises five medical images 4210 to 4250. Each medical image provides a 2D representation of a part of the human body, for example in a cross-sectional or axial view. The medical images 4210 to 4250 sample body tissue 4260 (shown as a mass for illustrative purposes only). The set of images is arranged according to a standard z-axis (not shown). The medical image 4220 shows a case where it actually represents two body tissues 4270 and 4280 that belong to the same body tissue 4260. By determining the intersection between a pair of bounding boxes (e.g., the bounding boxes enclosing the segments on images 4220 and 4230 respectively), determination S420, and merging S440 leading to the result of the bounding box enclosing the segments of the images (e.g., the segments of images 4220 and 4230), it can be determined that the two body tissues 4270 and 4280 belong to the same body tissue 4260.

[0067] An example of a method with iterative determination of intersections across pairs of bounding boxes will be described next.

[0068] This method converts the segmentation into individual lesions, and it is understood that this method can be applied to any human tissue. Next, this method iteratively processes all pairs of lesions to check whether they intersect. If they intersect, they are merged together. If they do not intersect, proceed to other lesions. After this method has completed the comparison of all lesions, this method checks that there are no new intersections in the newly merged lesions. Therefore, this method loops through all the lesions. Such an aggregation process ends when no more merges occur. At that point, this method outputs the lesions.

[0069] Referring to FIG. 5, an example of the iterative determination S40 is shown.

[0070] Iterative determination S40 starts (5000), and the acquired segment is selected as a 2D segmentation (5010) identified by applying a trained neural network to a set of medical images, for example (S20). The method converts the 2D segmentation into a 3D segment (also referred to as a "3D lesion") (5020), thereby giving the segment a 3D position (5030). Here, the term lesion is used in relation to medical images acquired to measure potential increases or decreases in lesions. However, any type of human tissue can be used. The purpose of converting the 2D segment into a 3D segment is to provide an artificial thickness to the 2D segment. In the example, the thickness can be obtained by calculating a bounding box surrounding the 2D segment, and the thickness of the bounding box is at least equal to the distance between two consecutive images in the set of images. It is understood that this distance is the same for pairs of consecutive images in the set of medical images. The thickness of the bounding box can be equal to the interval between two consecutive images in the set of images. For example, if the resolution of the imaging device generating the set of medical images is 1 mm (i.e., the distance along the Z-axis between two consecutive images (slices) in the set is 1 mm), the thickness of the bounding box for obtaining the 3D lesion is 1 mm. Interestingly, the calculation (acquisition) of the 3D bounding box is cost-effective in terms of computational resources and memory. Experience has shown that using the resolution of the imaging device as the thickness of the bounding box gives the best results.

[0071] The iteration can continue in the generation step for all 3D segment (or 3D lesion) pairs (5040), which defines segments that are obtained from the application of the trained neural network or can be merged after the iteration.

[0072] This method obtains a pair of segments (or "lesion pair" l1, l2) (or the next lesion pair in subsequent iterations) in the first iteration (5050). This method can determine whether the segments intersect by performing an intersection determination (S410) and subsequently performing an intersection determination between the pair of segments (S420) (5060). If the segments intersect, this method can proceed to merge segments l1 and l2 by calculating the resulting bounding box (S430) (5070). If the segments do not intersect in S420, this method maintains the pair of bounding boxes enclosing each segment. This method determines whether all pairs for which bounding boxes were generated intersect by determining whether there is a non-empty intersection between the resulting pairs of bounding boxes (5080). If there is a non-empty intersection (5090), this method can repeat steps 5040 and subsequent steps. If there is no non-empty intersection, this method determines that no merging has occurred between all segments (5090), and this method ends iteration S40 (5100).

[0073] Here, an example of detecting intersections between segments will be described.

[0074] Referring to FIG. 5, an example of determining the intersection between a pair of bounding boxes, subsequently determining the intersection between 2D bounding boxes, and then determining the intersection between segments enclosed by the pair of bounding boxes is shown.

[0075] The example of FIG. 6 starts from (6000) two 3D segments (also called "3D lesions" described with reference to FIG. 5) l1 and l2. The method determines (6010) the intersection between the pair of bounding boxes. If the intersection (6020) is empty, the method determines (6040) that there is no lesion intersection and the process ends. If the intersection of the pair (6020) is not empty, the method obtains (6030) the segments provided in the intersecting bounding boxes. The method determines (6060) whether the segments provided in the intersecting bounding boxes are less than a predetermined threshold max_distance away from a reference axis (in this case the z-axis), for example by comparing information regarding the separation of the corresponding images along the reference axis. max_distance can be selected according to the resolution of the scan that generates the image. max_distance can be selected such that it is at most equal to the interval between two consecutive images of a set of medical images. The predetermined threshold max_distance 6060 can be selected by performing a search for hyperparameters. Alternatively, the predetermined threshold max_distance 6060 can be selected as a function of the expected size of the lesion, for example 15 mm or more.

[0076] If the separation is below a predetermined threshold, the method calculates the 2D bounding boxes of each segment identified from at least two of the set of images. The method determines the intersection between at least two of the calculated 2D bounding boxes, for example by determining whether the 2D bounding boxes overlap (6070). If the intersection between the calculated 2D bounding boxes is empty, the method determines that there is no lesion intersection (6080) and ends the process. If the intersection between the calculated 2D bounding boxes is not empty, the method determines the intersection between the segments enclosed by the pair of bounding boxes, for example by obtaining the mask of each segment and determining whether the corresponding masks intersect (6090). If the intersection between the masks of the segments is not empty, the method merges the segments by calculating the resulting bounding box that encloses the segments. When two segments are merged, a new segment is created that includes the segmentations from both of the original lesions. The 3D bounding box is recalculated accordingly. And the previous lesions are discarded.

[0077] The method determines whether all segments have been inspected (6110). If not, the method returns to step 6050. Otherwise, the method ends. It should be understood that the masks of the segments are calculated / obtained as known in the art. For example, Mask R-CNN outputs an accurate segmentation mask for each segment.

[0078] Thus, the method detects as early as possible the case where the lesions do not intersect. The method transitions from a rough approximation to a fine calculation as the method progresses in the direction of calculating the intersection between the masks. That is, first, the method examines the 3D bounding boxes, then the method compares the 2D bounding boxes of each segmentation pair, and finally compares the 2D masks.

[0079] Figure 7 shows a pre-trained neural network.

[0080] This method may use a pre-trained 2D deep convolutional neural network (Mask R-CNN) 7020 trained to process CT scan images 7010.

[0081] The network receives as input a slice image 7011 and its 3D context and outputs zero or more detected 2D lesions 7030, for example from the liver 7031. Each detected 2D lesion is then either filtered or stacked together with other nearby detected lesions to create a 3D lesion. This method outputs RECIST measurements for each detected 3D lesion.

[0082] Figure 8 shows the inference of measurements in a RAW CT scan. This method may generate 3D segments 820 with measurements or stack them together as 3D lesions 830.

[0083] Next, the calculation of RECIST measurements will be described. Refer to Figure 9 showing the RECIST measurements calculated in segment 9000.

[0084] In this method, two specific diameters (major axis 9010 and minor axis 9020) of the segment of interest are identified. Since each segment needs to be measured at most once, in this method, it is confirmed that the segment selected to determine the measurement is the segment with the largest diameter. For this reason, in this method, the diameters of all slices constituting the segment are measured and only the maximum value is retained as the major axis.

[0085] To calculate the diameter, this method sets the following geometric approach. That is, in the case of the major axis 9010, in this method, the distances for all pairs of points on the contour are calculated and the maximum one is retained. In the case of the minor axis 9020, in this method, all points on the contour of the segment 9030 are examined and a line 9040 perpendicular to the major axis 9010 just calculated passing through this point is considered. In this method, the segment inside the contour is measured and the maximum value which is the line 9020 is retained.

Claims

1. A computer-implemented method for measuring human tissue from a set of medical images representing human tissue, comprising: obtaining a trained neural network configured to output segments of human tissue from the set of medical images (S10); applying the trained neural network to the set of medical images (S20) to thereby identify one or more segments of human tissue for at least two images of the set; calculating a bounding box surrounding each segment (S30); determining an intersection between pairs of bounding boxes (S410); when the intersection of the pair is not empty, determining an intersection between the segments surrounded by the pair of bounding boxes (S420); when the intersection between the segments is not empty, merging the segments by calculating a resulting bounding box surrounding the segments (S430); measuring the size of the segments comprised in the resulting bounding box (S50); and a method comprising the above.

2. Before determining the intersection between the segments, calculating, for each identified segment, a 2D bounding box from the at least two images of the set, each 2D bounding box surrounding the identified segment; determining an intersection between at least two of the calculated 2D bounding boxes; The method according to claim 1, further comprising: the determination of the intersection between the segments is performed only for segments for which the intersection between the 2D bounding boxes is not empty.

3. The method according to claim 1 or 2, further comprising: repeatedly determining (S40) the intersection (S410) across pairs of bounding boxes by repeating the determination (S420) and the merging (S430) for each remaining pair of bounding boxes.

4. The method according to any one of claims 1 to 3, further comprising: maintaining a pair of bounding boxes surrounding each segment when the intersection between the segments is empty.

5. Further comprising obtaining a distance between the images comprising the segment surrounded by the pair, wherein the determination (S420) of the intersection between the segments surrounded by the pair is performed only when the distance between the images comprising the segment is less than a predetermined threshold value. The method according to any one of claims 1 to 4.

6. The trained neural network is further configured to output a tag for identifying a segment of human tissue, and the determination of the intersection across the pair of bounding boxes is performed on segments having the same tag. The method according to any one of claims 1 to 5.

7. The determination of the intersection between the segments obtaining a mask for each segment, calculating the intersection between the obtained masks and the merging of the segments is performed only on segments for which the intersection between the obtained masks is not empty. The method according to any one of claims 1 to 6.

8. The trained neural network is further configured to output a 2D bounding box surrounding the segment of human tissue, and the calculation of the 2D bounding box is performed by the trained neural network. The method according to any one of claims 2 to 7.

9. Measuring the size of the segment further includes selecting a segment having the longest diameter provided in the resulting bounding box. The method according to any one of claims 1 to 8.

10. The set of medical images comprises a set of CT-SCAN images, a set of MRI images, a set of PET scan images, or a set of ultrasound images. The method according to any one of claims 1 to 9.

11. The human tissue represented by the set of medical images corresponds to a lesion of an organ, or an aneurysm or tissue of an organ. The method according to any one of claims 1 to 10.

12. A method for identifying the progression of human tissue, comprising: obtaining a current set of medical images representing a patient's human tissue; using the method according to any one of claims 1 to 11 to obtain a current measurement of the size of the segment; obtaining a measurement obtained from a past set of medical images representing the patient's human tissue; Calculating a difference between the current measurement value and the past measurement value, thereby identifying the progression of the human tissue A method comprising the above.

13. When a program is executed by a computer, a computer program comprising instructions for causing the computer to perform the method according to any one of claims 1 to 11 and / or the method according to claim 12.

14. A computer-readable storage medium on which the computer program according to claim 13 is recorded.

15. A system comprising a processor coupled to a memory on which the computer program according to claim 13 is recorded.

Images