Image segmentation system

Abstract

Disclosed herein is a medical system 100, 300 that includes a display 112 and a user interface 114. Execution of the machine-executable instructions 120 causes the processor 104 to receive (200) three-dimensional medical image data of an anatomical structure 128, receive (202) a three-dimensional segmentation 124 having one or more reference locations 800, display (204) at least one two-dimensional slice 126 of the three-dimensional medical image data, render (206) a cross-section 134 of the three-dimensional segmentation, provide (208) a control element 130 of the user interface that receives one-dimensional positions of at least one reference location along a predetermined one-dimensional path 806, receive (210) a one-dimensional position 137 from the control element, adjust (212) the three-dimensional segmentation 124 using the one-dimensional position, and update (214) the rendering of the cross-section of the three-dimensional segmentation.

Description

[Technical Field]

[0001] This invention relates to three-dimensional medical imaging, and more particularly to the segmentation of three-dimensional medical images. [Background technology]

[0002] Various medical imaging techniques, such as magnetic resonance imaging (MRI), computed tomography (CT), and three-dimensional ultrasound imaging, can segment three-dimensional image data to provide quantitative measurements of anatomical structures.

[0003] U.S. Patent Application No. 2013 / 0050207A1 discloses a method for navigating 3D images, which includes accessing a three-dimensional (3D) image dataset, generating a 3D mesh corresponding to 3D segmentation results using the 3D image dataset, and displaying a 3D surface rendering of 3D image intensity on the 3D mesh.

[0004] The academic paper Valenzuela et al., "FISICO: Fast Image Segmentation Correction," PLoS ONE 11(5), (2016)e0156035. https: / / doi.org / 10.1371 / journal.pone.0156035 discloses a 3D shape correction for image segmentation using 2D interaction. The correction process is performed using contour mapping. The user can select any point on the contour, and after the operation, the new shape is calculated using a deformation method. [Overview of the project]

[0005] The present invention provides a magnetic resonance imaging system, a computer program product, and a method as described in an independent claim.

[0006] One of the challenges with 3D segmentation systems is that adjusting the segmentation can be difficult. For example, while the 3D mesh used to represent the segmentation is conveniently visualized in an easily understandable manner on a 2D screen, it can be difficult to visualize the 3D segmentation and the measured 3D image data simultaneously. The current method for examining 3D image data and comparing it to 3D segmentation involves displaying slices of the 3D image data along with the cross-sections of the displayed 3D segmentation.

[0007] Segmentation includes parameters such as reference locations, which are points that identify anatomical landmarks, such as points or planes of anatomical structures. Embodiments provide means for adjusting the location of reference locations outside the displayed slice or moving reference locations outside the displayed slice. Embodiments provide control elements, such as sliders or dials, that allow input of the one-dimensional position of reference locations along a one-dimensional path. The user can manipulate the control elements, and the rendering of the cross-section of the three-dimensional section is updated. The one-dimensional path can be defined within the model or within the three-dimensional segmentation. In some cases, the three-dimensional segmentation can function as a model that defines the geometric relationship between the three-dimensional segmentation and the reference locations.

[0008] Several existing methods for correcting 3D segmentation use cross-sections of medical images and segmentation on a 2D plane. The system operator can select contour points of the segmentation and move them within the 2D plane. Correcting 3D segmentation in this way presents several problems. First, the position on the 2D plane must be selected very carefully.

[0009] For example, when adjusting cardiac segmentation, the vertices of the heart may be incorrectly located inside or outside the segmentation. To effectively adjust segmentation using only a two-dimensional plane, the system user must not only select a plane that passes through the correct locations of the vertices, but also select the locations of the vertices defined within the segmentation.

[0010] The embodiment can avoid this problem by providing a control element that receives the one-dimensional position of at least one reference location along a predetermined one-dimensional path. The one-dimensional path does not need to be within a two-dimensional slice in which the cross-section and segmentation of the medical image are viewed. This has the advantage that the three-dimensional segmentation can be properly adjusted without the need to precisely position the two-dimensional slice. This has the benefit of making the user interface more ergonomic. To properly edit the three-dimensional segmentation, the user checks the two-dimensional slice and adjusts the control element so that the segmentation best matches the medical image displayed in the two-dimensional slice. This reduces the amount of user-side configuration and data input required to correctly adjust the three-dimensional segmentation.

[0011] Another problem with adjusting 3D segmentation by directly manipulating contours within a 2D plane is that after moving the 2D points defining the contours of the 3D plane, the model automatically updates, rendering the current 2D plane obsolete for editing the segmentation. For example, a system user might spend considerable time properly positioning the 2D plane for editing the 3D segmentation. If the 3D segmentation is updated using a deformation method, the current 2D plane may become obsolete for subsequent editing. In that case, the user needs to reposition the 2D plane.

[0012] Using a control element that receives a one-dimensional position along a predetermined one-dimensional path eliminates or reduces the need to repeatedly reposition the two-dimensional slices and three-dimensional segmentation of three-dimensional medical imaging data during three-dimensional segmentation adjustment. This provides a more ergonomic user interface.

[0013] In one embodiment, the present invention provides a medical system including a display and a user interface. The medical system further includes a memory containing machine-executable instructions. The medical system further includes a processor that controls the medical system. Execution of the machine-executable instructions causes the processor to receive three-dimensional medical image data describing an anatomical structure. Execution of the machine-executable instructions further causes the processor to receive three-dimensional segmentation of the anatomical structure. The 3D segmentation may be received from different locations. In one example, it may be obtained from memory.

[0014] In another example, the data may be obtained from a storage device or from a remote location via a network interface. In yet another example, the 3D segmentation may be received from a user interface. In yet another example, an automated segmentation algorithm provides the 3D segmentation after the 3D medical image data has been received by a processor. The 3D segmentation includes one or more reference locations. As used herein, a reference location is a defined location on the 3D segmentation. This is, for example, a significant anatomical landmark or location identified within the 3D segmentation.

[0015] The execution of machine-executable instructions further causes the processor to display at least one two-dimensional slice of three-dimensional medical image data using a display. Two-dimensional slices of three-dimensional medical image data can be provided in various ways. In some cases, three-dimensional medical image data is acquired in slices. In other examples, three-dimensional medical image data is a complete three-dimensional dataset, and the two-dimensional slice is a cross-section of this three-dimensional data.

[0016] The execution of machine-executable instructions further causes the processor to render cross-sections of the 3D segmentation within at least one 2D slice on the display. As a result, the location of the reference point can be moved regardless of whether it is within the slice being displayed.

[0017] The execution of a machine-executable instruction further causes the processor to provide a user interface control element on the user interface for at least one reference location. The control element receives a one-dimensional position of at least one reference location along a given one-dimensional path. The execution of a machine-executable instruction further causes the processor to receive the one-dimensional position from the control element.

[0018] The execution of machine-executable instructions further causes the processor to adjust the 3D segmentation using the 1D position. The execution of machine-executable instructions further causes the processor to update the rendering of the cross-section of the 3D segmentation in at least one 2D slice on the display.

[0019] In this embodiment, the control element provides means for moving the position of at least one reference location. The control element is provided to do this. However, the movement of at least one reference location is not arbitrary, but along a predetermined one-dimensional path. This predetermined one-dimensional path is defined with respect to a three-dimensional segmentation. This is beneficial because it provides means for adjusting important parameters of the three-dimensional segmentation that are not included in one of the displayed two-dimensional slices. This is even more beneficial because it provides means for moving the reference location away from the displayed two-dimensional slice.

[0020] Moving at least one reference point along a predetermined one-dimensional path also provides a means of adjusting the three-dimensional segmentation in an anatomically meaningful way. This is not simply an arbitrary movement of a point, but an adjustment of gross parameters that can affect the view of all two-dimensional slices. For example, the user can manipulate a control element, which is updated in rendering and can be adjusted to improve the overall segmentation.

[0021] The medical systems used herein can take different forms in various examples. In one example, the medical system is, for example, a workstation in a radiology department or other medical center where medical professionals examine three-dimensional medical image data. In another example, the medical system is a remote computing system used for processing large amounts of medical imaging data. In yet another example, the medical system is a workstation or other computer system integrated with and controlling a medical imaging system. For example, the medical system may include a magnetic resonance imaging system, an ultrasound system, or even a computed tomography system.

[0022] The medical system can also display a variable number of 2D slices. In one example, only one slice is displayed, but even in this case, the user can improve segmentation by manipulating control elements. In other examples, multiple slices are displayed.

[0023] In another embodiment, the one-dimensional path is defined within a three-dimensional segmentation. This is useful because the implementation can adapt to existing data. In this case, the one-dimensional path moves in a meaningful way to the anatomical structures described in the three-dimensional medical image data.

[0024] The fact that a one-dimensional path is defined within a three-dimensional segmentation means that the one-dimensional path is part of a model used to define the three-dimensional segmentation. For example, the segmentation is defined by one or more anatomical landmarks. For example, the one-dimensional path passes through one or more anatomical landmarks. In another example, the one-dimensional path is defined by a curve or spline that fits one or more anatomical landmarks.

[0025] The fact that a one-dimensional path is defined in a three-dimensional segmentation means that the definition includes specifying the location of the one-dimensional path in the three-dimensional path.

[0026] In another embodiment, the three-dimensional segmentation is adjusted by calculating a vector movement of a reference location using a one-dimensional position. The three-dimensional segmentation is further adjusted by inputting the vector movement into a three-dimensional editing engine to update the three-dimensional segmentation. This embodiment is beneficial because software used to perform a complete three-dimensional edit of the segmentation can be used when the displayed two-dimensional projection does not represent the entire three-dimensional space represented by the three-dimensional segmentation. For example, changes in position along the path are useful for generating the vector movement.

[0027] In another embodiment, the vector movement of the reference location is input into the three-dimensional editing engine as the movement of a virtual mouse. In this example, the input to the three-dimensional editing engine is a simulated mouse movement or an operation of the segmentation. This also has the advantage that existing software for performing a three-dimensional edit of the segmentation can be used when the three-dimensional view is not displayed.

[0028] In another embodiment, the three-dimensional segmentation is a heart segmentation. This embodiment is beneficial because the heart of a subject has very clearly defined reference locations that need to be optimized and adapted for the heart to perform an accurate segmentation.

[0029] In another embodiment, at least one reference location includes the left ventricular apex.

[0030] In another embodiment, at least one reference location includes the right ventricular apex.

[0031] In another embodiment, at least one reference location includes the ventricular apex.

[0032] In another embodiment, at least one reference location includes the mitral valve surface.

[0033] In another embodiment, at least one reference location includes a tricuspid valve surface.

[0034] In another embodiment, the three-dimensional segmentation includes the valve surface.

[0035] In another embodiment, at least one reference location includes the above combination, provided they are not mutually exclusive.

[0036] In another embodiment, multiple two-dimensional slices of three-dimensional medical image data are perpendicular to the long axis of the left ventricle. For example, at least one reference location is the left ventricular apex. In this case, segmentation can be adjusted very effectively in each of the two-dimensional slices.

[0037] In some cases, the 2D slice is perpendicular (at a 90° angle) to the long axis of the left ventricle. This is because, after cardiac segmentation, the 3D medical image data is reformatted based on anatomical information. In another scenario, the slice is nearly perpendicular because it is the plane of the first slice acquired for the 3D medical image data. For example, the acquisition is set up so that the slice shows a short-axis cross-section through the heart. However, in the second case, the alignment is rough and some misalignment may still remain.

[0038] In another embodiment, the three-dimensional segmentation is prostate segmentation. This embodiment is beneficial because the anatomical shape of the prostate is very clearly defined. Prostate segmentation is effectively modified by modifying at least one reference location.

[0039] In another embodiment, at least one reference location includes the base of the prostate.

[0040] In another embodiment, the prostate segmentation further includes the prostate apex.

[0041] In another embodiment, the prostate segmentation further includes the location of the intermediate glandular surface of the prostate.

[0042] In another embodiment, prostate segmentation is a combination of any of the above-described prostate segmentation reference locations.

[0043] In another embodiment, the reference location includes a vertex.

[0044] In another embodiment, the vertex is two or more vertices.

[0045] In another embodiment, the reference location includes a triangle. In some variations, the reference location consists of multiple triangles.

[0046] In another embodiment, the reference location includes an anatomical landmark.

[0047] In another embodiment, the reference location includes a plane. For example, the reference location includes a single vertex or a set of vertices. Similarly, only a single triangle or a set of triangles may be specified. Likewise, multiple anatomical landmarks and planes may be reference locations.

[0048] In another embodiment, at least one two-dimensional slice is multiple two-dimensional slices. Some of the multiple two-dimensional slices are displayed simultaneously. This is beneficial because it provides a more effective means of moving at least one reference location to the appropriate location in order to optimize segmentation.

[0049] In another embodiment, the graphical user interface control unit is located outside the rendering of the 3D segmentation cross-sections. This embodiment has the advantage of ensuring that the 3D segmentation can be adjusted while minimizing the need to position the 2D cross-sections.

[0050] In another embodiment, the user interface further receives modifications to the cross-sections of a three-dimensional segmentation within at least one two-dimensional slice on the display. For example, the segmentation provides contours or projections that are placed on the display. For example, the user can drag, move, or manipulate these displayed cross-sections.

[0051] The execution of machine-executable instructions further causes the processor to receive cross-section modifications from the user interface. The execution of machine-executable instructions further causes the processor to adjust the 3D segmentation using the cross-section modifications. This involves, for example, updating other parts of the model at specific locations or in the vicinity of the moved portion.

[0052] In another embodiment, the execution of machine-executable instructions further causes the processor to provide 3D segmentation by segmenting 3D medical image data using an image segmentation module. For example, after the processor receives 3D medical image data, the data is input to, for example, an automatically functioning image segmentation module. The image segmentation module in various examples may be one of many different types. Some non-limiting examples include anatomical diagram-based segmentation modules, deformable shape segmentation modules that optimize fit and have energy constraints, as well as region-based segmentation modules, threshold-based segmentation modules, edge-based segmentation modules, and even neural networks trained to perform segmentation.

[0053] For example, a neural network receives 3D image data and outputs segmentation accordingly. As described herein, the system is used to improve or modify segmentation performed using one of these automated methods.

[0054] In another embodiment, the medical system further includes a medical imaging system that acquires three-dimensional medical image data from an imaging zone. The execution of machine-executable instructions is further configured to control the medical imaging system to acquire the three-dimensional medical image data. This embodiment is beneficial because it provides an integrated system that acquires medical image data and then provides improved segmentation.

[0055] In another embodiment, the medical imaging system is a magnetic resonance imaging system.

[0056] In another embodiment, the medical imaging system is a computed tomography system.

[0057] In another embodiment, the medical imaging system is an ultrasound imaging system that provides three-dimensional medical image data.

[0058] In another embodiment, the present invention provides a computer program product comprising machine-executable instructions for execution by a processor controlling a medical system. For example, the computer program product is housed in memory, a storage device, or a non-temporary storage medium. The medical system includes a display and a user interface. Execution of the machine-executable instructions causes the processor to receive three-dimensional medical image data describing anatomical structures.

[0059] The execution of a machine-executable instruction further causes the processor to receive a three-dimensional segmentation of an anatomical structure. The three-dimensional segmentation includes one or more reference locations. The execution of a machine-executable instruction further causes the processor to display at least one two-dimensional slice of the three-dimensional medical image data using a display. The execution of a machine-executable instruction further causes the processor to render a cross-section of the three-dimensional segmentation within at least one two-dimensional slice on the display.

[0060] The execution of the machine-executable instruction further causes the processor to provide a user interface control element on the user interface for at least one reference location. The control element receives a one-dimensional position of at least one reference location along a given one-dimensional path. The execution of the machine-executable instruction further causes the processor to receive the one-dimensional position from the control element. The execution of the machine-executable instruction further causes the processor to adjust the three-dimensional segmentation using the one-dimensional position. The execution of the machine-executable instruction further causes the processor to update the rendering of the cross section of the three-dimensional segmentation in at least one two-dimensional slice on the display.

[0061] In another embodiment, the execution of machine-executable instructions further causes the processor to save the updated 3D segmentation.

[0062] In various examples, control elements can take different forms. When using a control element to control the position of at least one reference location along a given one-dimensional path, any control element capable of providing one-dimensional coordinates may be used. For example, a slider, dial, or numerical input may be used. In other examples, a single control element can be used to input data to two or more reference locations. For example, if there are two reference locations, an XY pad or plane can be used to adjust two of them simultaneously in one dimension.

[0063] In another embodiment, the present invention provides a method for operating a medical system, the medical system including a display and a user interface. The method includes receiving three-dimensional medical image data articulating an anatomical structure. The method further includes receiving a three-dimensional segmentation of the anatomical structure, the three-dimensional segmentation including one or more reference locations. The method further includes displaying at least one two-dimensional slice of the three-dimensional medical image data using the display. The method further includes rendering a cross-section of the three-dimensional segmentation within at least one two-dimensional slice on the display.

[0064] The method further includes the step of providing a user interface control element on the user interface for at least one reference location. The control element receives a one-dimensional position of at least one reference location along a given one-dimensional path. The method further includes the step of receiving the one-dimensional position from the control element. The method further includes the step of adjusting the three-dimensional segmentation using the one-dimensional position. The method further includes the step of updating the rendering of the cross section of the three-dimensional segmentation in at least one two-dimensional slice on the display.

[0065] It is understood that one or more of the above embodiments of the present invention can be combined, provided that the combined embodiments are not mutually exclusive.

[0066] As will be recognized by those skilled in the art, aspects of the present invention can be embodied as apparatus, methods, or computer program products. Accordingly, aspects of the present invention can take the form of complete hardware embodiments, complete software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware embodiments, all of which are generally referred to as “circuits,” “modules,” or “systems.” Furthermore, aspects of the present invention can take the form of computer program products embodied in one or more computer-readable media in which computer executable code is embodied.

[0067] Any combination of one or more computer-readable media can be used. A computer-readable media is a computer-readable signal medium or a computer-readable storage medium. As used herein, “computer-readable storage medium” includes any tangible storage medium capable of storing instructions executable by the processor of a computing device. A computer-readable storage medium is also called a computer-readable non-temporary storage medium. A computer-readable storage medium is also called a tangible computer-readable medium. In some embodiments, a computer-readable storage medium can also store data accessible by the processor of a computing device. Examples of computer-readable storage media include, but are not limited to, floppy disks, magnetic hard disk drives, solid-state hard disks, flash memory, USB thumb drives, random-access memory (RAM), read-only memory (ROM), optical disks, magneto-optical disks, and processor register files. Examples of optical disks include compact disks (CDs) and digital versatile disks (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R disks. The term computer-readable storage medium also refers to various types of recording media that can be accessed by computer devices via a network or communication link. For example, data can be retrieved via a modem, the Internet, or a local area network. Computer executable code embodied in a computer-readable medium can be transmitted using any medium, including but not limited to wireless, wire, fiber optic cable, RF, or any suitable combination thereof.

[0068] Computer-readable signaling media may include propagated data signals in which computer-executable code is embodied, such as a portion of a baseband or carrier wave. Such propagated signals can take various forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. Computer-readable signaling media may be any computer-readable medium that can communicate, propagate, or transfer programs used in or in connection with an instruction execution system, apparatus, or device, rather than computer-readable storage media.

[0069] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that can be directly accessed by the processor. "Computer storage" or "storage" is an example of a computer-readable storage medium. Computer storage is any non-volatile computer-readable storage medium. In some embodiments, computer storage is also computer memory, and vice versa.

[0070] As used herein, “processor” includes electronic components capable of executing programs, machine-executable instructions, or computer-executable code. References to computing devices that include a “processor” should be interpreted as potentially including multiple processors or processing cores. For example, a processor may be a multi-core processor. A processor may also refer to a collection of processors within a single computer system, or a collection of processors distributed across multiple computer systems. The term computing device should also be interpreted as potentially referring to a collection or network of computing devices, each containing one or more processors. Computer-executable code may be executed by multiple processors within the same computing device, or it may be distributed across multiple computing devices.

[0071] Computer executable code may include machine executable instructions or programs that cause a processor to perform aspects of the present invention. Computer executable code for performing the operations of aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java®, Smalltalk®, and C++, or conventional procedural programming languages ​​such as the C programming language or similar languages, and may be compiled into machine executable instructions. In some cases, computer executable code may be in a high-level language or compiled form and used with an interpreter that generates machine executable instructions on the fly.

[0072] Computer executable code can run entirely on a user's computer, partially on a user's computer, as standalone software, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN). Alternatively, a connection to an external computer may be established (for example, via the Internet using an Internet service provider).

[0073] Aspects of the present invention are described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It is understood that each block or part of a block in a flowchart, diagram, and / or block diagram can, where applicable, be implemented in the form of computer executable code by computer program instructions. Furthermore, combinations of blocks in different flowcharts, diagrams, and / or block diagrams can be combined if they are not mutually exclusive. Such computer program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine such that the instructions are executed via the processor of the computer or other programmable data processing device to create means for implementing functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0074] These computer program instructions are also stored in a computer-readable medium that can instruct a computer, other programmable data processing device, or other device to function in a particular way, thereby generating a product containing instructions that implement functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0075] Computer program instructions can also be loaded into a computer, another programmable data processing device, or another device to cause the computer, the other programmable device, or the other device to execute a series of operational steps, thereby generating a computer-implemented process, in which the instructions executed on the computer or other programmable device provide a process for implementing a function / action specified in one or more blocks of a flowchart and / or block diagram.

[0076] As used herein, “user interface” refers to an interface that enables a user or operator to interact with a computer or computer system. A “user interface” is also called a “human interface device.” A user interface can provide information or data to an operator, or receive information or data from an operator. Using a user interface, a computer can receive input from an operator and provide output from the computer to the user. In other words, a user interface can be used to enable an operator to control or operate a computer, or an interface can be used to enable a computer to demonstrate the effects of the operator's control or operation. Displaying data or information on a display or graphical user interface is one example of providing information to an operator. Receiving data via a keyboard, mouse, trackball, touchpad, pointing stick, graphics tablet, joystick, gamepad, webcam, headset, pedal, wired glove, remote control, and accelerometer are all examples of user interface components that enable the reception of information or data from an operator.

[0077] As used herein, “hardware interface” includes interfaces that enable a computer system’s processor to interact with and control external computing devices and / or equipment. Hardware interfaces enable the processor to send control signals and commands to external computing devices and / or equipment. They also enable the processor to exchange data with external computing devices and / or equipment. Examples of hardware interfaces include, but are not limited to, universal serial buses, IEEE 1394 ports, parallel ports, IEEE 1284 ports, serial ports, RS-232 ports, IEEE-488 ports, Bluetooth® connectivity, wireless local area network connectivity, TCP / IP connectivity, Ethernet® connectivity, control voltage interfaces, MID interfaces, analog input interfaces, and digital input interfaces.

[0078] As used herein, “display” or “display device” includes output devices or user interfaces adapted for displaying images or data. Displays can output visual, auditory, or tactile data. Examples of displays include, but are not limited to, computer monitors, television screens, touchscreens, tactile electronic displays, braille screens, cathode ray tubes (CRTs), storage tubes, bistable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VFs), light-emitting diode (LED) displays, light-emitting displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, projectors, and head-mounted displays.

[0079] Magnetic resonance (MR) imaging data is defined herein as a measurement of radio frequency signals emitted from atomic spins using the antenna of a magnetic resonance apparatus during a magnetic resonance imaging scan. Magnetic resonance data is an example of medical image data. Magnetic resonance imaging (MRI) images, or MR images, are defined herein as a reconstructed two-dimensional or three-dimensional visualization of anatomical data contained in magnetic resonance imaging data. MR images are an example of three-dimensional medical image data. This visualization is performed using a computer. [Brief explanation of the drawing]

[0080] Preferred embodiments of the present invention are described below as merely examples, with reference to the drawings.

[0081] [Figure 1] Figure 1 shows an example of a healthcare system. [Figure 2] Figure 2 shows a flowchart illustrating how to operate the medical system shown in Figure 1. [Figure 3] Figure 3 shows a further example of a healthcare system. [Figure 4] Figure 4 shows a flowchart illustrating how to operate the medical system shown in Figure 3. [Figure 5] Figure 5 shows the adjustment of 3D segmentation using control elements. [Figure 6] Figure 6 shows the implementation of multiple control elements as sliders within the GUI. [Figure 7] Figure 7 shows the one-dimensional path corresponding to the slider in Figure 6. [Figure 8] Figure 8 shows the modification of 3D segmentation using a slider. [Modes for carrying out the invention]

[0082] Elements with the same number in these diagrams are equivalent elements or perform the same function. Elements already described are not necessarily described in later diagrams if their function is equivalent.

[0083] Figure 1 shows an example of a medical system 100. The medical system is shown as including a computer 102. The computer 102 includes a processor 104. The processor 104 is intended to represent one or more computing cores located in one or more locations. Thus, the processor 104 may be multiple processor cores and / or chips, and may be located in physically different computer systems in different locations. The processor 104 is connected to an optional hardware interface 106. If the medical system 100 includes additional components, the hardware interface 106 is present and used by the processor 104 to control them. The processor 104 is also connected to an optional user interface 108 and memory 110. Memory 110 is intended to represent any type of memory accessible by the processor 104. The user interface 108 is shown as including a display 112. The display has a graphical user interface 114.

[0084] Memory 110 is shown as containing machine-executable instructions 120. These machine-executable instructions 120 include instructions that enable the processor 104 to perform various control tasks, data processing tasks, and image processing tasks. Memory 110 is further shown as containing three-dimensional medical image data 122 and segmentation 124. Segmentation 124 includes segmentation of anatomical structures. The graphical user interface 114 has a window showing a rendering of two-dimensional slices 126 of the three-dimensional medical image data 122. Within the slices 126, anatomical structures 128 are shown.

[0085] The graphical user interface 114 is also shown as including a control element 130. In this example, the control element 130 is a slider. Box 132 represents the slider in a first position. The dashed line 134 represents a cross-section of the three-dimensional segmentation 124. In this example, the cross-section 134 does not fit well to the anatomical structure 128. In this case, the user moves the slider to a second slider position 136. The segmentation is then updated so that it now becomes the dotted line 138. This is the adjusted cross-section 138 of the three-dimensional segmentation 124.

[0086] When the slider 130 is moved from the first position 132 to the second position 136, a one-dimensional position 137 is stored in memory 110. This one-dimensional position 137 is used to update the segmentation by moving at least one reference location along a predetermined one-dimensional path.

[0087] Figure 2 shows a flowchart illustrating how to operate the medical system in Figure 1. First, in step 200, three-dimensional medical image data 122 describing the anatomical structure is received. Next, in step 202, three-dimensional segmentation 124 of the anatomical structure is received. The three-dimensional segmentation includes one or more reference locations that are not visible in the rendering of the two-dimensional slices 126. Therefore, the user cannot adjust them without the slider 130. Alternatively, the user can use the slider to move the reference locations from any of the displayed two-dimensional slices 126.

[0088] Next, in step 206, the cross section 134 of the 3D segmentation is rendered within the 2D slice 126. Then, in step 208, a control element 130 is provided on the user interface 114. In this example, only one reference location is provided. Other models may have additional sliders or additional control units. Next, the user adjusts the slider from the first position 132 to the second position 136, and in step 210, the 1D position is received from the control element 130. Then, in step 212, the 3D segmentation is adjusted using this 1D position. Then, in step 214, the rendering of the cross section 138 on the 2D slice 126 is updated.

[0089] Figure 3 shows a further example of the medical system 300. The medical system 300 in Figure 3 is similar to the medical system in Figure 1, except that it additionally includes a magnetic resonance imaging system 302. The display 112 is also part of the medical system 300, but is not shown in Figure 3. Figure 3 is intended to be representative. The magnetic resonance imaging system 302 can also be replaced with other types of medical imaging systems, such as computed tomography systems or ultrasound imaging systems.

[0090] The magnetic resonance imaging system 302 includes a magnet 304. The magnet 304 is a superconducting cylindrical magnet having a bore 306 within it. Different types of magnets can also be used; for example, both a split cylindrical magnet and a so-called open magnet can be used. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat is divided into two sections, allowing access to the magnet's isoplane. Such magnets can be used, for example, in combination with charged particle beam therapy. An open magnet has two magnet sections, one above the other, with sufficient space between them to accommodate a patient. The arrangement of the two sections is similar to that of a Helmholtz coil. Open magnets are popular because the patient is not as confined. Inside the cylindrical magnet cryostat is an assembly of superconducting coils.

[0091] Within the bore 306 of the cylindrical magnet 304 is an imaging zone 308 where the magnetic field is strong and uniform enough to perform magnetic resonance imaging. Within the imaging zone 308 is indicated a region of interest 309. Typically, the magnetic resonance data to be acquired is obtained for the region of interest. The subject 318 is shown supported by the subject support 320 so that at least a portion of the subject 318 is within the imaging zone 308 and the region of interest 309.

[0092] Within the region of interest 309, there is an anatomical structure 322. In this example, the anatomical structure 322 in subject 318 is the subject's heart. Other organs or structures, such as the prostate, can also be imaged.

[0093] Within the magnet's bore 306 is also a set of field gradient coils 310 used to acquire preliminary magnetic resonance data for spatially encoding the magnetic spins within the imaging zone 308 of the magnet 304. The field gradient coils 310 are connected to a field gradient coil power supply 312. The field gradient coils 310 are intended to be representative. Typically, the field gradient coils 310 contain three separate sets of coils for spatial encoding in three orthogonal spatial directions. The field gradient power supply provides current to the field gradient coils. The current supplied to the field gradient coils 310 is controlled as a function of time and is either sloped or pulsed.

[0094] Adjacent to the imaging zone 308 is a radio frequency coil 314 that manipulates the orientation of magnetic spins within the imaging zone 308 and receives radio transmissions from spins within the imaging zone 308. The radio frequency antenna may include multiple coil elements. The radio frequency antenna is also called a channel or antenna. The radio frequency coil 314 is connected to a radio frequency transceiver 316. The radio frequency coil 314 and the radio frequency transceiver 316 may be replaced by separate transmitting and receiving coils, as well as separate transmitters and receivers. The radio frequency coil 314 and the radio frequency transceiver 316 are understood to be representative. The radio frequency coil 314 is also intended to represent a dedicated transmitting antenna and a dedicated receiving antenna. Similarly, the transceiver 316 may represent separate transmitters and receivers. The radio frequency coil 314 may have multiple transmitting / receiving elements, and the radio frequency transceiver 316 may have multiple receiving / transmitting channels. For example, when parallel imaging techniques such as SENSE are performed, the radio frequency coil 314 has multiple coil elements.

[0095] The transceiver 316 and the gradient controller 312 are shown as being connected to the hardware interface 106 of the computer system 102.

[0096] Memory 110 is shown to include an additional pulse sequence command 330. The pulse sequence command enables the processor 104 to control the magnetic resonance imaging system to acquire magnetic resonance imaging data 332. The magnetic resonance imaging data 332 may be reconstructed by the processor into three-dimensional medical image data 122. Memory 110 is also shown to optionally include an image segmentation module 334. The image segmentation module 334 takes three-dimensional medical image data 122 as input and outputs three-dimensional segmentation 124 accordingly. As mentioned above, there are various means to achieve this.

[0097] Figure 4 shows a flowchart illustrating the operation method of the medical system 300 in Figure 3. The method in Figure 4 begins at step 400. In step 400, machine-executable commands 120 control the medical imaging system (in this case, the magnetic resonance imaging system 302) to acquire three-dimensional medical image data 122. In this specific example, pulse sequence commands 330 are used to acquire magnetic resonance imaging data 332, which is then reconstructed into three-dimensional medical image data 122. Next, in step 402, the three-dimensional medical image data 122 is input to the image segmentation module 334 to provide three-dimensional segmentation 124. The method then proceeds to step 200 in Figure 2, and the remainder of the method in Figure 4 follows the method shown in Figure 2.

[0098] Some examples may involve interactive 3D mesh editing along the normal / out-of-plane direction of the view plane, controlled by an additional GUI control unit.

[0099] A potential problem with 3D editing tools in MR applications is that only slice-by-slice editing is available in the user interface. This often means that, for example, in cardiac MR or prostate MR, the results can only be edited in a plane. In cardiac MR, if the short-axis plane is shown, it is not possible to change the apical height or valve surface. As an example, the left ventricle (LV) is segmented too short; that is, the segmented apex is above the actual apical slice. In this case, the mesh is not even visible in the actual apical slice. If the mesh contour is not displayed, it is not possible to edit the apex to the correct location in the image.

[0100] In another example, the adjustment of 3D segmentation can be divided into two components. In-plane interaction with the 3D mesh can still be performed using the mouse. Editing along the normal direction is performed by combining anatomical information from the 3D mesh with a 1D slider GUI control element or other control elements. The starting point and direction of editing are defined from the anatomical context of the model. For example, to move the apex, the position of the apex becomes the starting point, and the connecting line between the apex and the mitral valve defines the direction. As the slider is moved, the editing algorithm is supplied with information as if the mouse had dragged the position of the apex along that direction. In this way, the mesh is updated as if the apex had been dragged upward / downward.

[0101] Overall, this allows for control over in-plane and out-of-plane editing through a consistent editing toolkit that preserves the 3D mesh.

[0102] In some examples, interaction editing can be divided into two components. In some examples, in-plane interaction with the 3D mesh can be performed using the mouse. Editing with (non-negligible) components in the normal direction can be performed by combining anatomical information from the 3D mesh with a one-dimensional slider GUI or other control elements. Editing does not need to be precisely aligned with the normal direction. The starting point and direction of editing can be defined from the anatomical context of the model. For example, to move the apex, the position of the apex becomes the starting point, and the connecting line between the apex and the mitral valve defines the direction. When the slider is moved, the editing algorithm is supplied with information as if the mouse had dragged the position of the apex along that direction. In this way, the mesh is updated as if the apex had been dragged upward / downward.

[0103] Figure 5 shows a specific example where the anatomical structure is the heart. Four images are shown. The images in the top row represent short-axis images that are rendered and displayed to the subject. These are examples of 2D slices on the user interface. The image below, 502, is an image along the long axis of the heart and is not rendered, but is shown here for illustrative purposes. The first column 510 shows the original segmentation cross section 504 before correction. Column 512 shows the updated segmentation after a small slider correction. The endo apex begins to move into the slice. Column 514 shows the updated image segmentation 506 after a large slider correction that moves the contour of the endo apex to the desired height. Column 516 shows further correction after manual slice editing in the plane. The apex slider is then used again to adjust the latest mesh with respect to the apex height.

[0104] The behavior of control element 130 can be modified in different examples. For example, in one example, after releasing the slider and updating the mesh, slider 130 is reset directly to the center position, from which new editing can begin.

[0105] In another case, the slider remains in the target location even after being released, allowing the user to perform the following actions: - Check the accuracy of the current mesh position by examining other anatomical locations (e.g., scrolling through slices). - Further adjust the slider. This then modifies the previous slider editing step (i.e., using the same initial mesh as the previous step, but with a different target position along the one-dimensional axis).

[0106] In this example, the slider will reset to its center position with the next manual edit, or when any other slider that may exist is modified. Then, the next slider movement will perform the edit based on the mesh that is currently active, and all landmarks will be recalculated from the mesh.

[0107] Figure 6 shows an example of a portion of the graphical user interface 114. Again, this example is for a heart, and there are three sliders for correcting the positions of the apex, mitral valve, and tricuspid valve TV.

[0108] Figure 7 shows the model changes along each of the three one-dimensional paths shown in Figure 6. Path 700 is the path of the LV apex. Path 702 is the one-dimensional path of the tricuspid valve. Finally, path 704 is the path of the mitral valve. As each of these sliders is moved, the segmentation automatically adjusts to this change and the rendering is updated.

[0109] Figures 7 and 8: The left image shows an example of the slider control section in the GUI. The user clicks and moves the slider, which moves the structure within the mesh. Figure 8 shows an example of a structure that benefits from slider editing (LV apex 700, valve surfaces 702, 704). Here, the movement is shown in the longitudinal view. If the longitudinal view is unavailable or editing is not possible in the longitudinal view, the slider allows editing along these directions.

[0110] Examples include one or more of the following components: - Display Unit: Displays cardiac MR images with an overlay of segmentation results (i.e., segmented 3D mesh). An example of the display is shown in Figure 6. Here, the MR image is shown. Typically, only the short-axis image (top row) is shown. - Interactive 3D Editing Unit: Allows interaction with 3D meshes by clicking and directly dragging the displayed image / mesh. Typical usage: - The user views the image with the segmentation results overlaid to identify the area to edit. - The user clicks in the center of the area to be edited and presses the mouse button down. - The user drags the mouse to the desired location, and the displayed mesh deforms dynamically during this process. That is, the area around the starting point moves towards the new mouse position. The mesh is deformed in 3D space. - The user releases the mouse button at the desired location. - The deformed mesh is displayed and used for further analysis, such as volume calculations. -As the description indicates, you can edit the mesh in the direction corresponding to mouse movement on the displayed plane. If only the short-axis cross-section is shown, for example, you can move an LV wall inward and outward, but not up and down. -Slider editing unit: Allows you to modify the mesh along directions that are not possible with the indicated short-axis cutting plane. This includes: - A control unit for the graphical user interface corresponding to anatomically defined mesh correction. Typically, this is a slider control unit. See Figures 6 and 7. Preferred applications for cardiac MR segmentation editing are as follows: - Move the LV apex up and down - Move the mitral valve surface up and down. - Move the tricuspid valve surface up and down. - An anatomical unit that converts interaction with a slider control into virtual mouse movement. These virtual mouse movements are understood by the 3D editing engine, but cannot be directly executed by the user because there is no view plane. - From the anatomical context of the segmentation model, the position of the cardiac apex p apex and the mitral valve p MV is determined. - The direction n slider of the correction (virtual mouse movement) is calculated as a normalized difference vector - Interactive editing is initialized as if the user had clicked on p apex - Next, the user adjusts the slider value d slider . This determines the length of the cardiac apex edit (virtual mouse movement). As a result, the target mouse point is p current = p apex + d slider * n slider and is calculated as. This point p current is supplied to the 3D editing engine as if the user had moved the mouse there. - Note: In practice, the direction of n slider can be reversed so that the ventricle lengthens with a positive slider value.

[0111] In some examples, only parallel (or nearly parallel) slices are displayed. For example, a long-axis view (or other view) is presented, but editing is only possible in other views (i.e., short-axis views). Therefore, the fact that a particular view itself does not generally exist is not a requirement for the example to be valid.

[0112] Figure 8 illustrates the concept of slider editing. Point 800 represents the location of the vertex of the left ventricle. Line 802 represents the segmentation. Point 808 represents the in-plane position of the mitral valve 808. The dashed line 806 is a one-dimensional path defined between point 800 and point 808. Next, the one-dimensional path 806 is defined by the segmentation 808. As the slider moves, a one-dimensional position 810 is defined along the path 806. This can be changed to a vector movement 804 that can be used to modify the segmentation 802.

[0113] The present invention is illustrated and described in detail in the drawings and the above description, but such illustrations and descriptions should be considered illustrative or exemplary and not limiting. The present invention is not limited to the disclosed embodiments.

[0114] Other variations of the disclosed embodiments may be understood and implemented by those skilled in the art in carrying out the claimed invention, based on a review of the drawings, disclosures, and appended claims. In the claims, the word “including” is not exclusive of other elements or steps, and the singular is not exclusive of plural. A single processor or other unit may perform the functions of several items described in the claims. The mere fact that certain means are described in different dependent claims does not mean that combinations of these means cannot be used advantageously. Computer programs may be stored / distributed on any suitable medium, such as optical storage media or solid-state media, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems. Any reference numerals in the claims should not be construed as limiting the scope. [Explanation of Symbols]

[0115] 100 Healthcare Systems 102 Computer 104 Processors 106 Hardware Interfaces 108 User Interface 110 memory 112 displays 114 Graphical User Interface 120 Machine Executable Instructions 122 3D medical image data 124 3D Segmentation 126 Rendering of 2D slices 128 Anatomical Structure 130 Control elements (sliders) 132 First slider position 134 Cross-section of 3D segmentation 136 Second slider position 137 1D position 138 Adjusted cross-sections of 3D segmentation 200 Receive 3D medical image data describing anatomical structures. 202 Receive 3D segmentation of anatomical structures. 3D segmentation includes one or more reference locations. Use a 204 display to show at least one 2D slice of 3D medical image data. 206 Render a cross-section of a 3D segmentation within at least one 2D slice on the display. 208 Provide a user interface control element for at least one reference location on the user interface. 210 Receive one-dimensional position from control element 212 Adjust 3D segmentation using 1D position 214 Update the rendering of the cross-section of the 3D segmentation in at least one 2D slice on the display. 300 healthcare systems 302 Magnetic Resonance Imaging System 304 Magnet 306 Magnet Bore 308 Imaging Zone 309 Areas of Interest 310 Magnetic field gradient coil 312 Magnetic field gradient coil power supply 314 Radio frequency coil 316 Transceiver 318 subjects 320 Subject support 322 Anatomical Structure 330 Pulse Sequence Command 332 Magnetic Resonance Imaging Data 334 Image Segmentation Module 400 Control medical imaging systems to acquire 3D medical image data. 402 Segment 3D medical imaging data using the Image Segmentation Module 500 rendered short-axis images (2D slices) 502 Unrendered long-axis image 504 Original segmentation cross section 506 Adjusted segmentation section 510 Before slider correction 512 After small slider correction 514 After large slider correction 516 After manual mouse editing within the plane Paths to 700 vertices 702 Tricuspid valve pathway 704 Mitral valve pathway Location of 800 vertices 802 Segmentation 804 Vector Movement 806 1D Path 808 Location of the mitral valve 810 1D location

Claims

1. The display and User interface and Memory containing machine-executable instructions, A processor for controlling a medical system, wherein the machine executable instructions are provided to the processor, Receiving three-dimensional medical image data describing anatomical structures, Receiving the three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation includes one or more reference locations that identify one or more anatomical landmarks, and receiving the three-dimensional segmentation, Using the aforementioned display, at least one two-dimensional slice of the three-dimensional medical image data is displayed, Rendering the cross-section of the three-dimensional segmentation within the at least one two-dimensional slice on the display, The user interface is provided with a control element for adjusting the one-dimensional position of at least one of the reference locations along a predetermined one-dimensional path passing through one or more anatomical landmarks, wherein the predetermined one-dimensional path is a path defined with respect to the three-dimensional segmentation, and the control element receives the adjustment of the one-dimensional position of the reference location along the predetermined one-dimensional path with respect to the three-dimensional segmentation. Receiving the one-dimensional position of the reference location adjusted from the control element, The three-dimensional segmentation is adjusted by calculating the vector movement of the reference location using the adjusted one-dimensional position of the reference location and inputting the vector movement into the three-dimensional editing engine. A processor that causes the display to update the rendering of the cross-section of the three-dimensional segmentation within the at least one two-dimensional slice, A healthcare system, including the medical system.

2. The medical system according to claim 1, wherein the one-dimensional path is defined within the three-dimensional segmentation.

3. The medical system according to claim 1, wherein the vector movement of the reference location is input to the 3D editing engine as the movement of a virtual mouse.

4. The medical system according to any one of claims 1 to 3, wherein the three-dimensional segmentation is cardiac segmentation, and the reference location includes one of the left ventricular apex, right ventricular apex, ventricular apex, mitral valve surface, tricuspid valve surface, valve surface, and any combination thereof.

5. The medical system according to any one of claims 1 to 3, wherein the three-dimensional segmentation is prostate segmentation, and the reference location includes one of the locations of the prostatic base, the prostatic apex, the intermediate glandular surface of the prostate, and combinations thereof.

6. The medical system according to any one of claims 1 to 3, wherein the reference location further includes one of a vertex, a triangle, a plane, and a combination thereof.

7. The medical system according to any one of claims 1 to 3, wherein the at least one two-dimensional slice is a plurality of two-dimensional slices, and some of the plurality of two-dimensional slices are displayed simultaneously on the display.

8. The user interface further receives a modification of the cross-section of the three-dimensional segmentation in the at least one two-dimensional slice on the display, and the machine executable instruction further sends to the processor, Receiving the modification of the cross-section from the user interface, Adjusting the three-dimensional segmentation using the aforementioned modification of the cross-section, The medical system according to claim 7, which causes the system to update the rendering of the cross-section of the three-dimensional segmentation in each of the plurality of two-dimensional slices on the display.

9. The medical system according to any one of claims 1 to 3, wherein the execution of the machine-executable instructions further causes the processor to provide the three-dimensional segmentation by segmenting the three-dimensional medical image data using an image segmentation module.

10. The medical system according to any one of claims 1 to 3, further comprising a medical imaging system for acquiring the three-dimensional medical image data from an imaging zone, wherein the execution of the machine-executable instructions is further configured to control the medical imaging system to acquire the three-dimensional medical image data.

11. The medical system according to claim 10, wherein the medical imaging system is one of a magnetic resonance imaging system, a computed tomography system, and an ultrasound imaging system.

12. The medical system according to any one of claims 1 to 3, wherein the control element of the user interface for the reference location on the user interface includes one of a slider, a dial, a graphical user interface control unit, and a combination thereof, located outside the rendering of the cross section of the three-dimensional segmentation.

13. A computer program comprising machine-executable instructions for execution by a processor controlling a medical system, wherein the medical system includes a display and a user interface. The machine-executable instruction is given to the processor, Receiving three-dimensional medical image data describing anatomical structures, Receiving the three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation includes one or more reference locations that identify one or more anatomical landmarks, and receiving the three-dimensional segmentation, Using the aforementioned display, at least one two-dimensional slice of the three-dimensional medical image data is displayed, Rendering the cross-section of the three-dimensional segmentation within the at least one two-dimensional slice on the display, The user interface is provided with a control element for adjusting the one-dimensional position of at least one of the reference locations along a predetermined one-dimensional path passing through one or more anatomical landmarks, wherein the predetermined one-dimensional path is a path defined with respect to the three-dimensional segmentation, and the control element receives the adjustment of the one-dimensional position of the reference location along the predetermined one-dimensional path with respect to the three-dimensional segmentation. Receiving the one-dimensional position of the reference location adjusted from the control element, The three-dimensional segmentation is adjusted by calculating the vector movement of the reference location using the adjusted one-dimensional position of the reference location and inputting the vector movement into the three-dimensional editing engine. A computer program that causes the computer program to update the rendering of the cross-section of the three-dimensional segmentation within the at least one two-dimensional slice on the display.

14. A method for operating a medical system, wherein the medical system includes a display and a user interface, and the method is The steps include receiving three-dimensional medical image data describing anatomical structures, A step of receiving a three-dimensional segmentation of the anatomical structure, wherein the three-dimensional segmentation includes one or more reference locations that identify one or more anatomical landmarks, and the step of receiving the three-dimensional segmentation, The steps include: the display showing at least one two-dimensional slice of the three-dimensional medical image data; The steps include rendering the cross-section of the three-dimensional segmentation within the at least one two-dimensional slice on the display, The steps of providing a control element on the user interface for adjusting the one-dimensional position of at least one of the reference locations along a predetermined one-dimensional path passing through one or more anatomical landmarks, wherein the predetermined one-dimensional path is a path defined with respect to the three-dimensional segmentation, and the control element receives adjustments to the one-dimensional position of the reference location along the predetermined one-dimensional path with respect to the three-dimensional segmentation. The steps include receiving the one-dimensional position of the reference location adjusted from the control element, The steps include: adjusting the 3D segmentation by calculating the vector movement of the reference location using the adjusted 1D position of the reference location and inputting the vector movement into the 3D editing engine; The steps include updating the rendering of the cross-section of the three-dimensional segmentation within the at least one two-dimensional slice on the display, Methods that include...

Images