Methods and systems for real-time image 3D segmentation regularization
By combining regularized brushes and machine learning algorithms with medical image data, the segmentation results are updated in real time, which solves the problem of noisy data affecting segmentation accuracy and precision, and achieves more efficient medical image segmentation results.
Patent Information
- Application Number
- CN202310057651.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-25
- Filing Date
- 2023-01-19
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-01-19
AI Technical Summary
Existing medical image segmentation algorithms are prone to producing inaccurate results when faced with noisy medical image data, especially due to the influence of factors such as noise, motion, and artifacts, which leads to a decrease in segmentation accuracy and precision.
The initial segmentation of medical image data is updated in real time using user input based on a regularized brush. The probabilistic map is calculated by combining machine learning algorithms. The segmentation results are improved by combining user interaction and machine learning.
It improves the accuracy and precision of medical image segmentation, reduces the impact of noisy features on segmentation results, and allows referring physicians to manipulate segmentation boundaries more intuitively and achieve local smoothing and localization.
Smart Images

Figure CN116503315B_ABST
Abstract
Description
Technical Field
[0001] The implementation schemes of the subject matter disclosed herein relate to image segmentation, and more specifically to real-time 3D image segmentation regularization of medical image data. Background Technology
[0002] Clinical decisions can be derived from the analysis of one or more sets within a data source. In the field of radiology, this typically involves the analysis of regions of interest from medical image data, which may include 2D or 3D medical images such as images of organs (kidneys, liver, spleen, etc.), blood vessels, bones, etc. Medical image analysis may be performed at the request of the referring physician for a specific purpose; this purpose may include detecting, assessing, and / or monitoring the progression of anatomical abnormalities such as lesions, aneurysms, atrophy, etc.
[0003] To access these regions of interest in medical image data and perform the desired analysis, the data corresponding to these regions must first be accurately and robustly separated from other data. While many context-specific algorithms exist for segmenting regions of interest such as organs within medical image data, the variability of many medical data sources can lead to inaccurate results for other anatomical features or pathological structures such as lesions and nodules. Furthermore, factors within medical image data such as scale, noise, motion, partial volume, and other artifacts can hinder the accuracy and precision of segmentation algorithms. Summary of the Invention
[0004] In one implementation, a method for real-time segmentation regularization of medical data includes: updating an initial segmentation of the medical image data in real time based on user input to a regularization brush applied to the medical image data, the user input to the regularization brush allowing modification of the volume of the initial segmentation.
[0005] It should be understood that the above brief description is provided to introduce selected concepts further described in the detailed embodiments in a simplified form. This is not intended to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims following the detailed embodiments. Furthermore, the claimed subject matter is not limited to embodiments that address any shortcomings mentioned above or in any part of this disclosure. Attached Figure Description
[0006] This disclosure will be better understood by referring to the following description of non-limiting embodiments, in which:
[0007] Figure 1 A drawing view of an imaging system according to one embodiment is shown;
[0008] Figure 2 A block diagram of an exemplary imaging system according to one embodiment is shown;
[0009] Figure 3 A block diagram of an algorithm for real-time image segmentation is shown;
[0010] Figure 4 An exemplary method for updating image segmentation using user input from a regularized brush is shown;
[0011] Figure 5 An exemplary method for using a machine learning model to compute a probability map for image segmentation is shown;
[0012] Figure 6 It shows Figure 5 An exemplary operation of a machine learning model, in addition to generating ground truth segmentation based on a binary mask of the machine learning model's output from the organ representation, generates sample training data for training the machine learning model.
[0013] Figure 7 An exemplary user interface for image segmentation is shown; and
[0014] Figure 8 An exemplary incorrect segmentation of an image of a lung is shown, including a regularization brush that can be applied by the user to correct the image segmentation. Detailed Implementation
[0015] The following describes various implementations of systems and methods for real-time image segmentation of medical image data. Image segmentation may include dividing a digital image into different components (in other words, dividing a digital image into image segments, where each image segment is a collection of pixels) and may be used to simplify and / or transform the representation of the image into something more meaningful and easier to analyze (e.g., by a referring physician). During the analysis of medical image data, which may be in 2D or 3D, applying segmentation to the medical image data to allow the referring physician to easily characterize certain anatomical features (e.g., organs, blood vessels, bones, etc.) for diagnosis, analysis, etc., may be useful. In some examples, image segmentation may be performed automatically based on the medical image data and features to be segmented via, for example, machine learning algorithms. In other examples, segmentation may be performed via image analysis algorithms combined with user interaction with the medical image data.
[0016] An example of the latter type of segmentation (e.g., involving user input and algorithmic modifications to the segmentation) is through user interface tools such as edge-aware brush tools. Edge-aware brush tools can utilize medical image data to identify boundaries within the medical image data and allow users (e.g., radiologists) to interactively and in real-time modify the segmentation. Modifications to the segmentation can be achieved by the user viewing the segmentation overlaid on the medical image via an edge-aware brush, which is a user interface tool that allows the user to edit the segmentation within a sphere (in 3D, or a circle in 2D) surrounding the cursor, thus allowing the user to expand or shrink the segmentation within the sphere / circle. Expanding / shrinking the segmentation within the sphere of the edge-aware brush can be boundary-sensitive, allowing the boundary crossings within the medical image to be limited, as determined by the image processing algorithms operating on the medical image. However, in the case of noisy medical image data, such as medical image data containing unwanted image gradients, image artifacts that can cause misleading image boundaries, this can mislead user interaction tools such as edge-aware brushes that rely solely on the medical image data. Therefore, user-interactive tools can facilitate quick editing by users, but also allow for flexible handling of noisy features within medical image data as needed.
[0017] The regularization brush allows for user interaction with medical image data to provide segmentation masks. In addition to the current segmentation, the referring physician can apply the regularization brush with user input (such as the position of the cursor pointer on the medical image data, the radius of the sphere containing the cursor pointer, and options to expand / decrease / maintain the sphere's radius) to update the segmentation based on the user input by generating a new segmentation mask. This includes generating a binary mask to be used for updating the segmentation (to be discussed). Figure 2 Beyond the thresholding process (explained in more detail), generating a new segmentation mask based on the current segmentation and user input may include: preprocessing the current segmentation (including adaptive resampling of the segmentation based on user input from a regularization brush); and computing a probability map on the segmentation mask that indicates the probability that the generated new segmentation is the desired segmentation. In some examples, the probability map may be generated based on applying a low-pass filter (e.g., a mean filter, a Gaussian filter, an anisotropic filter, etc.) to the current segmentation mask. In other examples, the probability map may be computed based on machine learning techniques. Then, in response to user input, the segmentation updates described above can be iteratively updated continuously in real time.
[0018] In this way, by applying real-time image segmentation of medical data based on user input from a regularization brush and by incorporating machine learning algorithms to adapt the segmentation mask to the medical image data, referring physicians can interact with image segmentation in real time in an intuitive manner, reducing inter-user / inter-user variation. Compared to, for example, edge-aware brushes, the advantage of updating segmentation based on initial segmentation and user input is that noisy features in the medical image data (e.g., image noise, image artifacts such as beam hardening and scattering, and features such as tumors and Legionnaires' disease) do not obscure the segmentation performed by the regularization brush, allowing referring physicians to easily manipulate the segmentation. By performing regularization based on input from the segmentation and user input, referring physicians can more easily achieve local smoothing and localization of segmentation boundaries.
[0019] The medical image to be examined can be, for example, a radiographic image, and can be generated from one or more imaging modalities, including X-rays, ultrasound, MRI, etc. Image segmentation based on user input can then be performed on the medical image data. An exemplary imaging system could be, for example... Figure 1 The computerized computed tomography (CT) system 100 shown is illustrated. Figure 2 It shows something similar to Figure 2 A second exemplary imaging system 200 is provided for the CT system 100. 3D medical image data received from the exemplary imaging system can then be applied to it by a user (e.g., a radiologist) via image segmentation using an interactive algorithm. Figure 3 A block diagram of an exemplary algorithm for real-time image segmentation applicable to received 3D medical image data is provided. When applied to medical image data, user input may come from the use of a regularized brush. Figure 4 The document provides exemplary methods for updating image segmentation based on user input from a regularized brush. As described above, in some examples, updating the segmentation mask based on user application of a regularized brush may involve using a machine learning algorithm to compute a probability map indicating the likelihood that the generated new segmentation is the desired segmentation. Figure 4 The document provides an exemplary method for using machine learning algorithms to generate probabilistic graphs. Figure 5 An exemplary machine learning algorithm is shown for generating a probabilistic map of segments based on the generated training data; Figure 6 It shows Figure 5 An exemplary operation of the training process of a machine learning algorithm. Figure 7 An exemplary user interface for regularizing brushes is shown, while Figure 8 An exemplary application of regularized brushes to incorrect segmentation of medical image data is shown.
[0020] Figure 1An exemplary CT system 100 configured for CT imaging is shown. Specifically, the CT system 100 is configured to image a subject 112 (such as a patient, inanimate object, one or more manufactured parts) and / or foreign objects (such as dental implants, stents, and / or contrast agents present in the body). In one embodiment, the CT system 100 includes a gantry 102, which may further include at least one x-ray source 104 configured to project an x-ray radiation beam 106 (see [link to documentation]). Figure 2 This is used to image the subject 112 lying on the examination table 114. Specifically, the X-ray source 104 is configured to project an X-ray radiation beam 106 toward a detector array 108 positioned on the opposite side of the gantry 102. Although Figure 1 Only a single x-ray source 104 is depicted, but in some embodiments, multiple x-ray sources and detectors may be used to project multiple x-ray radiation beams 106 to acquire projection data corresponding to different energy levels of the patient. In some embodiments, the x-ray source 104 can achieve dual-energy gemstone spectral imaging (GSI) by rapidly switching peak kilovolt (kVp) voltage. In some embodiments, the x-ray detector used is a photon-counting detector capable of distinguishing x-ray photons of different energies. In other embodiments, two sets of x-ray sources and detectors are used to generate dual-energy projections, one set configured for low kVp and the other for high kVp. Therefore, it should be understood that the methods described herein can be implemented using both single-energy acquisition techniques and dual-energy acquisition techniques.
[0021] In some embodiments, the CT system 100 also includes an image processor unit 110 configured to reconstruct an image of the target volume of the subject 112 using iterative or analytical image reconstruction methods. For example, the image processor unit 110 may use an analytical image reconstruction method such as filtered back projection (FBP) to reconstruct an image of the patient's target volume. As another example, the image processor unit 110 may use iterative image reconstruction methods such as advanced statistical iterative reconstruction (ASIR), conjugate gradient (CG), maximum likelihood expectation maximization (MLEM), model-based iterative reconstruction (MBIR), etc., to reconstruct an image of the target volume of the subject 112. As further described herein, in some examples, in addition to iterative image reconstruction methods, the image processor unit 110 may also use analytical image reconstruction methods (such as FBP).
[0022] In some CT imaging system configurations, an X-ray source projects a cone-shaped X-ray beam, which is collimated to lie in the XYZ plane of a Cartesian coordinate system and is often referred to as the "imaging plane." The X-ray beam passes through the object being imaged, such as a patient or subject. After being attenuated by the object, the X-ray beam strikes an array of detector elements. The intensity of the attenuated X-ray beam received at the detector array depends on the attenuation of the beam by the object. Each detector element in the array generates a separate electrical signal, which is a measurement of the X-ray beam attenuation at the detector location. Attenuation measurements from all detector elements are acquired individually to produce a transmission distribution.
[0023] In some CT systems, a gantry is used to rotate the X-ray source and detector array around the object to be imaged within the imaging plane, causing the angle at which the radiation beam intersects the object to continuously change. A set of X-ray radiation attenuation measurements (e.g., projection data) from the detector array at a single gantry angle is referred to as a “view.” A “scan” of the object comprises a set of views acquired at different gantry angles or viewing angles during a single rotation of the X-ray source and detector. It is conceivable that the benefits of the methods described herein stem from medical imaging modalities beyond CT; therefore, as used herein, the term “view” is not limited to the uses described above regarding projection data from a single gantry angle. The term “view” is used to mean a data acquisition whenever multiple data acquisitions from different angles exist (whether from CT, positron emission tomography (PET), or single-photon emission CT (SPECT) acquisitions), and / or any other modality (including modalities yet to be developed) and their combinations in fusion implementations.
[0024] Projection data is processed to reconstruct an image corresponding to a two-dimensional slice obtained through an object, or, in some examples, to reconstruct an image corresponding to a three-dimensional rendering of the object, the projection data includes multiple views or scans. One method for reconstructing an image from a set of projection data is known in the art as filtered backprojection. Transmission and emission tomography reconstruction techniques also include statistical iterative methods such as maximum likelihood expectation maximization (MLEM) and ordered subset expectation reconstruction techniques, as well as iterative reconstruction techniques. This method converts attenuation measurements from scans into an integer called the “CT number” or “Henness unit,” which is used to control the brightness of the corresponding pixel on a display device.
[0025] To reduce overall scan time, a "spiral" scan can be performed. To perform a "spiral" scan, the patient is moved while data from a predetermined number of slices is acquired. Such systems generate a single spiral from a cone-beam spiral scan. The spiral mapped out by the cone beam produces projection data, from which an image in each predetermined slice can be reconstructed.
[0026] As used herein, the phrase "reconstructed image" is not intended to exclude embodiments of the invention in which data representing an image is generated rather than a visual image. Therefore, as used herein, the term "image" broadly refers to both a visual image and the data representing a visual image. However, many embodiments generate (or are configured to generate) at least one visual image.
[0027] Figure 2 It shows something similar to Figure 1 An exemplary imaging system 200 of a CT system 100. According to various aspects of this disclosure, the imaging system 200 is configured to examine a subject 204 (e.g., Figure 1Imaging is performed on the subject 112. In one embodiment, the imaging system 200 includes a detector array 108 (see [link to image processing system]). Figure 1 The detector array 108 also includes a plurality of detector elements 202 that together sense an X-ray radiation beam 106 passing through the subject 204 (such as a patient) (see [link to relevant documentation]). Figure 2 To acquire the corresponding projection data. Therefore, in one embodiment, the detector array 108 is manufactured in a multi-slice configuration including multiple rows of cells or detector elements 202. In such a configuration, another row or multiple rows of detector elements 202 are arranged in parallel for acquiring projection data.
[0028] In some embodiments, the imaging system 200 is configured to traverse different angular positions around the subject 204 to acquire the desired projection data. Therefore, the gantry 102 and the components mounted thereon may be configured to rotate about a center of rotation 206 to acquire projection data, for example, at different energy levels. Alternatively, in embodiments where the projection angle relative to the subject 204 varies over time, the mounted components may be configured to move along a generally curved path rather than along a circumference.
[0029] Therefore, as the X-ray source 104 and detector array 108 rotate, detector array 108 collects data of the attenuated X-ray beam. The data collected by detector array 108 then undergoes preprocessing and calibration to adjust the data to represent the line integral of the attenuation coefficient of the scanned subject 204. The processed data is typically referred to as the projection.
[0030] In some examples, individual detectors or detector elements 202 in detector array 108 may include photon counting detectors that register interactions of individual photons into one or more energy bins. It should be understood that the methods described herein can also be implemented using energy integration detectors.
[0031] The acquired projection dataset can be used for Base Material Decomposition (BMD). During BMD, the measured projections are converted into a set of material density projections. The material density projections can be reconstructed to form a pair or set of material density maps or images for each corresponding base material (such as bone, soft tissue, and / or contrast agent maps). The density maps or images can then be correlated to form a volumetric rendering of the base material (e.g., bone, soft tissue, and / or contrast agent) in the imaging volume.
[0032] Once reconstructed, the base material image generated by imaging system 200 reveals the internal features of subject 204 represented by the densities of the two base materials. Density images can be displayed to illustrate these features. In traditional methods of diagnosing medical conditions (such as disease states), and more generally, medical events, radiologists or physicians will consider a hard copy or display of the density image to identify features of interest. Such features may include lesions, size, and shape of specific anatomical structures or organs, as well as other features that should be identifiable in the image based on the individual practitioner's skill and knowledge.
[0033] In one embodiment, the imaging system 200 includes a control mechanism 208 for controlling the movement of components, such as the rotation of the gantry 102 and the operation of the x-ray source 104. In some embodiments, the control mechanism 208 further includes an x-ray controller 210 configured to provide power and timing signals to the x-ray source 104. Additionally, the control mechanism 208 includes a gantry motor controller 212 configured to control the rotational speed and / or position of the gantry 102 based on imaging requirements.
[0034] In some embodiments, control unit 208 further includes a data acquisition system (DAS) 214 configured to sample analog data received from detector element 202 and convert the analog data into digital signals for subsequent processing. DAS 214 may also be configured to selectively aggregate a subset of analog data from detector element 202 into a so-called macro detector, as further described herein. The data sampled and digitized by DAS 214 is transferred to a computer or computing device 216. In one example, computing device 216 stores the data in storage device 218. For example, storage device 218 may include hard disk drives, floppy disk drives, optical disc read / write (CD-R / W) drives, digital versatile optical disc (DVD) drives, flash memory drives, and / or solid-state storage drives.
[0035] Additionally, computing device 216 provides commands and parameters to one or more of the DAS 214, x-ray controller 210, and rack motor controller 212 to control system operation, such as data acquisition and / or processing. In some embodiments, computing device 216 controls system operation based on operator input. Computing device 216 receives operator input, such as commands and / or scan parameters, via an operator console 220 operatively coupled to computing device 216. Operator console 220 may include a keyboard (not shown), a mouse, and / or a touchscreen to allow the operator to specify commands and / or scan parameters.
[0036] Although Figure 2Only one operator console 220 is shown, but more than one operator console may be coupled to the imaging system 200, for example, for inputting or outputting system parameters, requesting checks, plotting data, and / or viewing images. Furthermore, in some embodiments, the imaging system 200 may be coupled via one or more configurable wired and / or wireless networks (such as the Internet and / or VPNs, wireless telephone networks, wireless LANs, wired LANs, wireless WANs, wired WANs, etc.) to multiple displays, printers, workstations, and / or similar devices, for example, located locally or remotely within an institution or hospital or in completely different locations.
[0037] In one implementation, for example, imaging system 200 includes or is coupled to a Picture Archiving and Communication System (PACS) 224. In an exemplary specific implementation, PACS 224 is further coupled to a remote system (such as a radiology information system, a hospital information system) and / or coupled to an internal or external network (not shown) to allow operators in different locations to supply commands and parameters and / or obtain access to image data.
[0038] The computing device 216 operates the inspection table motor controller 226 using operator-provided and / or system-defined commands and parameters. This inspection table motor controller, in turn, controls the inspection table 114, which may be an electric inspection table. Specifically, the inspection table motor controller 226 can move the inspection table 114 to properly position the subject 204 within the rack 102 to acquire projection data corresponding to the target volume of the subject 204.
[0039] As previously described, the DAS 214 samples and digitizes the projection data acquired by detector element 202. Subsequently, the image reconstructor 230 uses the sampled and digitized X-ray data to perform high-speed reconstruction. Although Figure 2 Image reconstructor 230 is shown as a separate entity, but in some embodiments, image reconstructor 230 may be part of computing device 216. Alternatively, image reconstructor 230 may not be present in imaging system 200, and alternatively, computing device 216 may perform one or more functions of image reconstructor 230. Furthermore, image reconstructor 230 may be located locally or remotely and may be operatively connected to imaging system 200 using wired or wireless networks. Specifically, one exemplary embodiment may use computing resources in a "cloud" network cluster for image reconstructor 230.
[0040] In one embodiment, image reconstructor 230 stores reconstructed images in storage device 218. Alternatively, image reconstructor 230 may transfer the reconstructed images to computing device 216 to generate usable patient information for diagnosis and evaluation. In some embodiments, computing device 216 may transfer the reconstructed images and / or patient information to a display or display device 232 communicatively coupled to computing device 216 and / or image reconstructor 230. In some embodiments, reconstructed images may be transferred from computing device 216 or image reconstructor 230 to storage device 218 for short-term or long-term storage.
[0041] The various methods and processes further described herein (such as those referenced below) Figures 3 to 6 The described methods may be stored as executable instructions in a non-transitory memory on a computing device (or controller) in the imaging system 200. In one embodiment, the computing device 216 may include instructions in the non-transitory memory and may apply the methods described herein at least partially to the reconstructed image after receiving it from the image reconstructor 230. In another embodiment, the methods and processes described herein may be distributed between the image reconstructor 230 and the computing device 216.
[0042] In one implementation, display 232 allows an operator to evaluate the anatomical structures of the image. Display 232 may also allow the operator to select the volume of interest (VOI) and / or request patient information, for example, via a graphical user interface (GUI), for subsequent scanning or processing.
[0043] In some implementations, imaging system 200 may utilize computing device 216 to segment 3D medical imaging data generated by the imaging system for use by a radiologist or physician via interaction with display device 232 and operator console 220. Computing device 216 may use a real-time regularization algorithm for initial segmentation of the 3D medical image data, thereby updating the segmentation in response to user input via a regularization brush in the user interface (UI). The user may update the initial segmentation by first drawing one or more views of the medical image data and then applying a regularization brush to a segmentation overlaid on the drawn medical image data. For example, user input to the regularization brush may include (as set by, for example, the position of the mouse on operator console 220) the position of the regularization brush, the radius of the influence sphere of the regularization brush, and adjustments to the radius of the influence sphere of the regularization brush. The algorithm may then compute a probability map based on the user input to the regularization brush and a low-pass filter applied to the segmentation. After computing the probability map, the algorithm can then further apply thresholding to the probability map (e.g., discarding voxels / pixels with probabilities below a certain level) to output an update of the segmentation within the influence sphere of the regularized brush, based on the thresholded probability map. The computation of the probability map and the subsequent thresholding applied by the user via the regularized brush allow for adaptive smoothing of the segmentation in a manner easily adjustable and interpretable by the user in real time.
[0044] In some implementations, the low-pass filter applied to the initial segmentation can be implemented using machine learning algorithms, particularly convolutional neural networks (CNNs). The CNN can receive a database of segmentation masks, such as those from a hard drive or generated by one or more annotation tools, as input, and can generate a set of modified segmentation masks that include one or more segmentation masks from the database of segmentation masks, in order to train the CNN to correctly filter out irrelevant artifacts from this set of modified segmentation masks. The CNN can then output a smooth probability map based on the training and the initial segmentation.
[0045] In this way, Figures 1 to 2 The system enables segmentation regularization for 3D medical image data, and the system includes: a computerized computed tomography (CT) system (e.g., Figure 1 The CT system is configured to generate 3D medical image data of the patient (100%); and a user interface (UI) (e.g., Figure 2 (as in 220 and 232), the UI is configured to allow users to segment 3D medical image data. The UI may include: a graphical user interface (GUI) including one or more segmentation tools; a regularization brush; and a computing device (e.g., ...). Figure 2(216) The computing device is communicatively coupled to a CT system and a UI, and can draw one or more views of 3D medical image data generated by the CT system. The computing device can then receive: a first user input from the GUI, the first user input including adjustments to the radius of a regularized brush influence sphere and the brush's push power; and a second user input from the regularized brush applied to the drawn 3D medical image data, the second user input including the brush's position; and an initial segmentation of the 3D medical image data. Based on the first and second user inputs, the computing device can then extract regions of interest (ROIs) from the drawn 3D medical image data; adaptively resample the drawn 3D medical image data within the ROIs; compute a probability map of the initial segmentation; apply thresholding to the probability map to generate a binary mask; and output the updated segmentation of the 3D medical image data based on the binary mask.
[0046] Figure 3 A block diagram of a first method 300 for real-time image segmentation is shown. In addition to medical image data 304, the first method 300 may take an initial segmentation 302 to be edited as input. The medical image data 304 may be an image or a collection of images generated by a medical imaging device (e.g., from...). Figure 1 CT system 100 or Figure 2 (Example imaging system 200 generates medical images). Medical image data 304 can be a 3D representation (e.g., composed of voxels) or a collection of 2D images (e.g., image slices). When the algorithm is first applied, the initial segmentation 302 can be the same as the segmentation 306 to be generated. The initial segmentation can be provided by a user (e.g., a radiologist) or can be generated from medical image data 304 via an algorithm (e.g., an image recognition algorithm, such as a convolutional neural network (CNN) trained to identify relevant features (such as organs, blood vessels, bones, etc.) from medical image data). Segmentation 306 can then be modified according to the modifications brought about by the regularization brush 312, which will be about Figure 4 To describe in more detail.
[0047] Image data 304 and segmentation 306 can be taken as input to perform real-time volume rendering 308 for display to a user. Real-time volume rendering 308 may include, but is not limited to, displaying a volume on image data 304 in a given plane, where the segmentation mask is an overlay (blended with a given transparency), and the volume is the rendering of the segmentation mask. Real-time volume rendering 308 may be a combination of 2D / 3D views for enhancing the visualization of the 3D mask. For example, volume rendering may include one or more of axial views, sagittal views, coronal views, oblique views (the above are 2D views), and / or volume rendering 3D views. Real-time volume rendering 308 may then await brush input 310 from the user, which may be input for real-time regularization brush 312 to generate an updated segmentation mask of segmentation 306. The inventors note that real-time volume rendering 308 and regularization brush 312 are performed in real time, thereby allowing immediate feedback from the user for accurate correction of segmentation 306.
[0048] Figure 4 This demonstrates the use of real-time regularized brushes (such as...) Figure 1 The block diagram of the second method 400 for generating output using the regularized brush 312). (Regarding...) Figures 1 to 2 The system described herein is used to illustrate the second method 400, but it should be understood that similar methods can be used with other systems without departing from the scope of this disclosure. The second method 400 can be implemented in a controller or computing device (e.g., Figure 2 Execute on 200).
[0049] The second method 400 can be as follows: Figure 3 A portion of the first method 300 described herein; specifically, the user input box 402 may have a similar... Figure 3 The brush input 310 has basically the same function, and the box 412 may have the same Figure 3 Segmentation 306 has essentially the same function as segmentation 410, which may have the same function as segmentation 306. Figure 3 The regularized brush 312 has essentially the same functionality, and the updated segmentation 424 can again have the same... Figure 3 The 306 segment is essentially similar in function, but the 306 segment is in Figure 3 The first method 300 is iteratively updated throughout its operation. The second method 400 may include a user input box 402, where user input includes the desired update of the segmentation mask (such as...) via a regularized brush. Figure 1 The user input field 402 may include the mouse position 404, the radius 406 of the regularized brush sphere, and an input 408 for expanding, decreasing, or maintaining the radius 406 of the regularized brush sphere. The user input field 402 may be accessed by the user based on data superimposed on medical image data (such as...). Figure 1 Volume rendering of image data 304 (such as image data 304) Figure 1 The regularized brush (including an interactive cursor on the image of a sphere with a regularized brush) is applied through user interaction on a real-time volume rendering (308). This defines where the brush will operate and how the segmented volume of the sphere will be updated (expanded, reduced, or maintained). The fourth input parameter of the regularized brush is the segmentation 410, which can be, for example, an initial segmentation of medical image data. The four input parameters can be fed into box 412 to generate the updated segmentation 424, which can be... Figure 1 The updated version of the 306 split.
[0050] Expanding, reducing, or maintaining the radius 406 of the regularized brush sphere can be linked to the user interface (to be discussed). Figure 5 (For a more detailed explanation of the user interface) shortcuts or different buttons on the mouse. For example, the left button, middle button, and right button can respectively expand, regularize, and reduce the segmentation. These refinement capabilities, such as the regularization brush's sphere radius of 406, are desired to allow for segmentation correction in a much shorter time.
[0051] Box 412 includes several sequential steps. The first step 414 of box 412 includes reducing the dimension of the selected region of interest (e.g., a portion of the volume drawing within the sphere of the regularized brush) by removing image data outside the sphere of the regularized brush (also known as the region of interest (ROI)). The second step 416 of box 412 includes adaptively resampling the volume drawing to resize the ROI, thereby ensuring that segmentation computation can be performed in real-time according to the controller's computational and memory constraints.
[0052] Following the adaptive resampling in the second step 416 of box 412, the third step 418 may include, for example, calculating mask regularization based on user input such as from user input box 402 to output a probability map that is more likely to be the desired segmentation. The output of the probability map computation may be an image with the same dimensions as the adaptively resampled volume, and each voxel has a given probability (for 3D volume segmentation; alternatively, the probability map computation may be an image with the same dimensions in the 2D case, and each pixel has a given probability), which takes a value between 0 and 1 (inclusive). For example, a value close to 1 for a given pixel / voxel in the probability map means that the given pixel / voxel is more likely to be part of the desired segmentation. In contrast, a value close to 0 means that the given pixel / voxel may be outside the desired segmentation.
[0053] The algorithm following the probability map computation in step 418 of box 412 can be an algorithm capable of applying a low-pass filter to a given segmentation mask. For example, any of the mean filter, Gaussian filter, anisotropic filter, and other standard filters can be applied to provide the probability map computation. The algorithm can also use more sophisticated methods for computation of the probability map, including machine learning techniques (to be discussed later). Figure 5 (To be further described), such as by using a convolutional neural network (CNN) to generate a probability map based on a training dataset with binary masks that have faults such as image noise, unexpected image features (e.g., tumors, lesions) and image artifacts (e.g., beam hardening).
[0054] After the initial probability map calculation in step 418, box 412 may proceed to step 420, which may include introducing a threshold into the probability map. The threshold may indicate a value of probability below which pixels / voxels of a given probability map may not be included in the desired segmentation. For example, by selecting a threshold of 0.5, the desired segmentation can be regularized without modifying the volume of the desired segmentation. However, if the user wants a larger desired segmentation, the selected threshold may be set to, for example, 0.3. Conversely, if the user wants a smaller desired segmentation, the threshold may be set to, for example, 0.7. The output of the thresholding process in step 420 of box 412 may be a binary image that can be applied to a segmentation mask, wherein portions of the resampled segmentation outside the binary image may not be included in the updated segmentation, and wherein portions of the resampled segmentation within the binary image may be included in the updated segmentation.
[0055] Following the thresholding process in the fourth step 420 of box 412, the second method 400 may proceed to the fifth step 422 to update the segmentation based on the probability map calculated in the third step 418 and the thresholding process in the fourth step 420. After updating the segmentation in the fifth step 422, the second method 400 may output the updated segmentation 424.
[0056] Figure 5 This illustrates the methods used to train machine learning models to compute probability graphs (such as...). Figure 4 The third method 500 (the probability graph calculation in the third step 418 of box 412). In some examples, the third method 500 can be used during the calculation of the probability graph as... Figure 4 The third method 500 is part of the third step 418 of the second method 400. The third method 500 may include a first part 502 of offline training of the machine learning model and a second part 508 of online computation for the probabilistic graph.
[0057] The first part 502 of the offline training of the third method 500 may include a large database 504 of binary segmentation masks in 3D and a machine learning model training part 506 based on this database. The binary segmentation masks included in the database 504 can be volumes composed of voxels with values of 1 (included in the segmentation) or 0 (not included in the segmentation). For example, a machine learning model may be trained to regularize the shape of a given organ (e.g., the liver). In one example, the database 504 of segmentation masks may be a segmentation of a single type of feature (such as the liver). In other examples, the database 504 of segmentation masks may include many segments of many different types of features (including several different types of organs, blood vessels, ligaments, bones, etc.).
[0058] Then, the offline training process of the first part 502 of the third method 500 can use the database 504 of segmentation masks for the organ, allowing the algorithm to learn the shape of the organ within the training part 506. During the training part 506, simulated changes to the database 504 of segmentation masks can be used to train the third method 500 to produce accurate probability maps, even in the presence of image artifacts. For example, the training part 506 may include applying several modifications to sample segmentation masks within the database 504 of segmentation masks to train a machine learning model to correctly filter out irrelevant artifacts from the segmentation mask data. For example, among other modifications to produce more accurate performance for the machine learning model, modifications to the segmentation mask in the database 504 of segmentation masks during training 506 may include: adding noise (adding isolated pixels / voxels or removing isolated pixels / voxels from the mask), generating holes within the segmentation mask (e.g., to train the machine learning model to fill the holes), generating ripples in the boundaries of the segmentation mask (e.g., to train the machine learning model to correct such ripples), adding / removing straight lines from the segmentation mask (e.g., to simulate beam hardening artifacts), and adding repeating patterns (e.g., to simulate a highly textured image).
[0059] The second part 508 of the algorithm can then be applied in step 510 to segmentation (such as...). Figure 3 The division of 306 and Figure 4 The segmentation (410) is used to generate a probability map. The probability map can be a smooth volume composed of voxels (in the 3D case) or pixels (in the 2D case), where each pixel / voxel can take a value between 0 (unlikely to be included in the segmentation) and 1 (likely to be included in the segmentation), including extreme values. The smooth probability map can then be thresholded, as previously described in... Figure 4 As described in step 420.
[0060] Figure 6 The machine learning model (such as about) is shown Figure 5 The exemplary operation 600 of the described third method 500 may include input to a binary segmentation mask 602, which may be, for example, a binary mask of an organ to be segmented. The binary segmentation mask 602 may be taken from a database of segmentation masks provided to the algorithm (such as...). Figure 5 Database 504). Additionally, the segmentation mask 602 can be a local mask, such that the size of the binary mask is within the algorithm's ROI (e.g., regarding...). Figure 4 Within the ROI described in the first step 414. Based on the binary segmentation mask 602, the machine learning model can then generate possible simulated inputs 604 to the binary segmentation mask 602 in order to provide training data to improve the machine learning model's ability to generate more accurate probability maps (such as in...). Figure 4 (In the probability graph calculation of the third step 418 of box 412). Exemplary modifications to the binary segmentation mask 602 shown in the simulated input 604 of the machine learning model include adding holes 606, boundary fluctuations 608, and adding / removing lines 610 and additive noise 612, as common examples. A large number of modifications can be applied to the binary segmentation mask 602; the examples above indicate several categories of such modifications.
[0061] In addition to generating possible simulated inputs 604 for the machine learning model to produce more accurate performance, the machine learning model can also generate an output 614 of the ground truth 616 of the binary segmentation mask 602. For example... Figure 6 As shown, the ground truth 616 is not a binary mask: the transition between inside and outside the segment can be highly smoothed. A machine learning model can learn to mimic these transitions, providing a probability map with a value of approximately 0.5 at the boundaries of the ground truth 616. By training the machine learning model offline via simulated input 604, this model allows for tuning of the regularized brush (such as regarding...). Figure 1 The regularized brush discussed here is used to more accurately reproduce the true segmentation mask of the object under consideration.
[0062] Figure 7 An exemplary user interface 700 is shown, including a segmentation tool for image annotation. In addition to other user inputs, the user interface 700 may include tools for regularizing brushes (such as those for...). Figure 3 The command for discussing regularized brushes.
[0063] The user interface for the regularization brush may include a swipe capability 702, thereby allowing the application of the regularization brush to the sphere (such as...) by influencing the cursor pointer of the regularization brush. Figure 4The sphere's size can be adjusted (expanded or reduced) by a given division within a radius of 406. The size of the sphere can be adjusted via the position of the adjusting slider 712, which may include decreasing the sphere's radius (by moving the slider to the left) or increasing the sphere's radius (by moving the slider to the right).
[0064] The push capability 702 allows for shrinking segments within the sphere of the regularized brush cursor pointer, as indicated in icon 708, or allows for expanding segments within the sphere of the regularized brush cursor pointer, as indicated in icon 710. Shrinking and expanding segments via the regularized brush can include previously defined ROIs as specified by the sphere. Figures 2 to 6 All the steps described in the document.
[0065] Furthermore, the segmentation can be changed in 3D mode 706 or modified in 2D 704, which can correspond to a 2D slice of the segment taken along a given plane. Modification of the segmentation in 2D can correspond to a given slice of medical image data (such as a 3D segmented slice from an axial view, sagittal view, coronal view, or oblique view), while modification of the segmentation in 3D can include modifying the 3D volume of the segmentation, the segmentation visible from the 2D slice, or from a 3D view (such as a volumetric 3D view).
[0066] Figure 8 An exemplary error segmentation 800 applied to an image 802 of the lung is shown, including a regularization brush (such as regarding...). Figure 3 The regularization brush discussed can be applied by the user to correct image segmentation. A sphere 804, representing the cursor pointer of the regularization brush, is shown above an incorrectly segmented image of a lung. After a mouse click, the regularization brush can be applied, resulting in the output of the corrected segmentation of the lung image.
[0067] In this way, by providing a real-time adaptive tool for image segmentation, more accurate segmentation based on user input (e.g., from a radiologist) can be achieved, making the segmentation more user-interpretable. This tool can be designed to correct a given segmentation derived from a file on a hard drive, from an automated algorithm from other annotation tools, or from other sources. During image segmentation, certain elements can alter the image and cause errors in the segmentation mask. Such elements can include: image noise, image artifacts, and tumors / lesions / high texture, etc. In particular, image noise can lead to noisy segmentation, holes or unnatural object boundaries in the segmentation, image artifacts such as beam hardening and scattering, which can mislead the algorithm and generate noisy segmentation, and tumors / lesions / high texture can introduce contours and textures into the image that are difficult to segment manually using traditional editing tools.
[0068] All of these elements can lead to unwanted image gradients and / or noisy image boundaries. Such elements can even erase the spacing between the object to be segmented and the surrounding image (e.g., COVID nodules in the lungs can have the same density as external tissue), thus blurring image information to guide annotation.
[0069] Other user-interactive tools for segmenting medical image data, such as edge-aware brushes, can use image analysis to determine boundaries between features in the image. Edge-aware brushes can include user input to edit the segmentation and provide modifications to the segmentation that are sensitive to image boundaries (e.g., edge-aware brush adjustments to the segmentation may not be willing to cross image boundaries), as determined by the edge-aware brush algorithm. However, as mentioned above, in cases where the image is noisy and / or the image gradients are unclear, edge-aware brushes may lead to undesirable adjustments to the segmentation based on noisy image data.
[0070] By relying on segmentation mask data, unlike relying solely on image data for segmentation mask updates, tools such as edge-aware brushes avoid elements from image data that can typically obscure segmentation mask updates. Furthermore, the location of segmentation boundaries can be inaccurate, and it may be desirable to move those boundaries. By incorporating user input from a regularized brush, the boundaries of the segmentation mask can be easily adjusted. The advantage of using user input from a regularized brush to update the segmentation mask is that it allows for rapid updates in a user-interpretable manner.
[0071] This disclosure provides support for a method for real-time segmentation regularization of medical image data, the method comprising: updating the initial segmentation of the medical image data in real time based on user input of a regularization brush applied to the initial segmentation of the medical image data, the user input of the regularization brush allowing modification of the volume of the initial segmentation. In a first example of the method, the user input of the regularization brush includes the position of the regularization brush on the medical image data, the radius of a sphere of the regularization brush, and an adjustment to the radius of the sphere of the regularization brush. In a second example of the method, the second example optionally includes the first example, and another input of the regularization brush includes the initial segmentation of the medical image data. In a third example of the method, the third example optionally includes one or both of the first and second examples, the update of the initial segmentation including extracting a region of interest (ROI) of the medical image data, the ROI being contained within the sphere of the regularization brush. In a fourth example of the method, the fourth example optionally includes one or more or each of the first to third examples, and the method further includes: applying adaptive resampling to the medical image data within the ROI. In a fifth example of the method, the fifth example optionally includes one or more or each of the first to fourth examples, and the method further includes: computing a probability map within the ROI, the probability map including voxels with values between 0 and 1, and wherein the probability map includes the probability of each voxel included in the probability map within the updated segmentation within the ROI. In a sixth example of the method, the sixth example optionally includes one or more or each of the first to fifth examples, populating the probability map by applying a low-pass filter to the initial segmentation within the ROI, the low-pass filter being one of a mean filter, a Gaussian filter, an anisotropic filter, and the output of a convolutional neural network (CNN). In a seventh example of the method, the seventh example optionally includes one or more or each of the first to sixth examples, the updated segmentation within the ROI being based on applying a thresholding process to the probability map. In an eighth example of the method, which optionally includes one or more or each of the first to seventh examples, the method further includes: outputting an updated segmentation of the medical image data, the updated segmentation being based on a change made to the initial segmentation within the ROI.
[0072] This disclosure also provides support for a method for real-time segmentation regularization of medical image data, the method comprising: updating an initial segmentation of the medical image data, the update comprising: drawing one or more views of the medical image data; receiving user input to a regularization brush applied to the drawn medical image data; calculating a probability map of the initial segmentation based on the user input to the regularization brush and a function for applying a low-pass filter to the initial segmentation; and outputting an updated segmentation of the medical image data based on the user input to the regularization brush and the calculated probability map. In a first example of the method, the user input includes: the position of the regularization brush, the radius of an influence sphere of the regularization brush, and an adjustment to the radius of the influence sphere of the regularization brush. In a second example of the method, the second example optionally includes the first example, wherein the low-pass filter applied to the initial segmentation is implemented via machine learning. In a third example of the method, the third example optionally includes one or both of the first and second examples, wherein the machine learning used in the low-pass filter is implemented via a convolutional neural network (CNN). In a fourth example of the method, the fourth example optionally includes one or more, or each of the first to third examples, and the CNN receives a database of segmentation masks as input. In a fifth example of the method, the fifth example optionally includes one or more, or each of the first to fourth examples, and the database of segmentation masks includes a set of segmentation masks from a hard drive or a set of segmentation masks generated by automated functions from one or more other annotation tools. In a sixth example of the method, the sixth example optionally includes one or more, or each of the first to fifth examples, and operating the CNN includes: generating a set of modified segmentation masks including one or more segmentation masks from the database of segmentation masks; training the CNN on the set of modified segmentation masks to correctly filter out irrelevant artifacts from the set of modified segmentation masks; and outputting a smoothed probability map based on the initial segmentation and the training of the CNN. In a seventh example of the method, which optionally includes one or more or each of the first to sixth examples, the updated segmentation of the medical image data is output by: applying a thresholding process to the smoothing probability map and updating a portion of the initial segmentation within the influence sphere of the regularization brush.
[0073] This disclosure also provides support for a system for segmentation regularization of 3D medical image data, the system comprising: a computerized computed tomography (CT) system configured to generate 3D medical image data of a patient; a user interface (UI) configured to allow a user to segment the 3D medical image data, the UI including: a graphical user interface (GUI) including one or more segmentation tools; a regularization brush; and a computing device communicatively coupled to the CT system and the UI, the computing device being configured with instructions in non-transitory memory that, when executed, cause the computing device to: draw one or more views of the 3D medical image data generated by the CT system; and receive first user input from the GUI, the first user input including the regularization brush. The system adjusts the radius of the influence sphere of the brush and the swivel ability of the regularized brush; receives a second user input from the regularized brush applied to the drawn 3D medical image data, the second user input including the position of the regularized brush; receives an initial segmentation of the 3D medical image data; extracts a region of interest (ROI) from the drawn 3D medical image data, the ROI being the influence sphere of the regularized brush; adaptively resamples the drawn 3D medical image data within the ROI; computes a probability map of the initial segmentation; applies a thresholding process to the probability map to generate a binary mask; and outputs an updated segmentation of the 3D medical image data based on the first and second user inputs of the regularized brush, the computed probability map, and the thresholding process applied to the probability map. In a first example of the system, the probability map for computing the initial segmentation is based on the first and second user inputs of the regularized brush and a function for applying a low-pass filter to the initial segmentation. In a second example of the system, the second example optionally includes the first example, and the adjustment of the push capability of the regularization brush includes expanding the segment by applying the regularization brush and reducing the segment by applying the regularization brush.
[0074] As used herein, elements or steps listed in the singular and beginning with the word "a" or "an" should be understood to not exclude a plurality of said elements or steps unless such exclusion is explicitly stated. Furthermore, references to "one embodiment" of the invention are not intended to be construed as excluding the existence of additional embodiments that also include the referenced features. Moreover, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" elements or multiple elements having a particular characteristic may include additional such elements that do not have that characteristic. The terms "comprise" and "in..." are used as concise linguistic equivalents to the corresponding terms "comprising" and "wherein". Furthermore, the terms "first," "second," and "third," etc., are used merely as notations and are not intended to impose numerical requirements or a particular order of position on their objects.
[0075] This written description uses examples to disclose the invention, including the best mode, and also enables those skilled in the art to practice the invention, including making and using any device or system and performing any included methods. The scope of patentability of the invention is defined by the claims and may include other examples that would occur to those skilled in the art. Such other examples are intended to fall within the scope of the claims if they have structural elements that are not indistinguishable from the literal language of the claims, or if they include equivalent structural elements that have minor differences from the literal language of the claims.
Claims
1. A method for real-time segmentation regularization of medical image data (304), the method comprising: The initial segment (302) of the medical image data (304) is updated in real time based on user input (310) of a regularization brush (312) applied to the initial segment (302) of the medical image data, wherein the user input of the regularization brush allows modification of the volume of the initial segment. The update of the initial segmentation (302) includes extracting the region of interest (ROI) of the medical image data, the ROI being contained within a sphere of the regularization brush (312). The user input (310) of the regularization brush (312) includes the position of the regularization brush on the medical image data, the radius of the sphere of the regularization brush, and the adjustment of the radius of the sphere of the regularization brush. The method further includes: Adaptive resampling is applied to the medical image data (304) within the ROI; Calculate a probability map within the ROI, wherein the probability map is populated by applying a low-pass filter to the initial segment 302 within the ROI, and wherein the probability map includes the probability that the voxels included in the probability map are within the updated segment (306) within the ROI; and Thresholding is applied to the probability map to generate a binary mask for updating the initial segmentation within the ROI.
2. The method of claim 1, wherein another input to the regularization brush (312) includes the initial segmentation (302) of the medical image data (304).
3. The method according to claim 1, wherein, The probability graph includes voxels with values between 0 and 1.
4. The method according to claim 3, wherein the low-pass filter is one of a mean filter, a Gaussian filter, an anisotropic filter, and the output of a convolutional neural network (CNN).
5. The method according to claim 1, further comprising: The updated segmentation (306) of the medical image data is output, which is based on the changes made to the initial segmentation (302) within the ROI.
6. A system for real-time segmentation regularization of medical image data (304), the system comprising: The controller stores instructions in non-transitory memory, and when these instructions are executed, the controller causes the controller to: Update the initial segmentation of the medical image data (302), the update including: Draw one or more views of the medical image data; User input (310) is received for a regularization brush (312) applied to drawn medical image data, wherein the update of the initial segmentation (302) includes extracting the region of interest (ROI) of the medical image data, the ROI being contained within a sphere of the regularization brush (312), wherein the user input includes: the position of the regularization brush (312), the radius of the influence sphere of the regularization brush, and an adjustment to the radius of the influence sphere of the regularization brush; Adaptive resampling is applied to the medical image data (304) within the ROI; The probability map within the ROI is computed based on the user input of the regularized brush and a function for applying a low-pass filter to the initial segmentation, wherein the probability map includes the probability that the voxels included in the probability map are within the updated segmentation (306) within the ROI. Thresholding is applied to the probability map to generate a binary mask for updating the initial segmentation within the ROI; and Output the updated segmentation of the medical image data, the updated segmentation being based on the user input of the regularized brush and the calculated probability map.
7. The system of claim 6, wherein the low-pass filter applied to the initial segmentation is implemented by machine learning, and wherein the machine learning used in the low-pass filter is implemented via a convolutional neural network (CNN).
8. The system of claim 7, wherein the CNN receives a database of segmentation masks as input, and wherein the database of segmentation masks comprises a set of segmentation masks from a hard disk drive or a set of segmentation masks generated by automated functions from one or more other annotation tools.
9. The system of claim 8, wherein the controller includes additional instructions to: Generate a set of modified segmentation masks that include one or more segmentation masks in the database of the segmentation masks; The CNN is trained on the set of modified segmentation masks to correctly filter out irrelevant artifacts from the set of modified segmentation masks; and Output a smooth probability map, which is based on the initial segmentation and the training of the CNN.
10. The system of claim 9, wherein outputting the updated segmentation (306) of the medical image data (304) comprises: Thresholding is applied to the smoothed probability map and a portion of the initial segmentation (302) within the influence sphere of the regularized brush (312) is updated.
Citation Information
Patent Citations
Dividing method based on CT image liver tumor focus
CN105488781A
