Automatic split input tool configuration

Abstract

A method is disclosed herein. The method includes receiving (200) a three-dimensional medical image (122) and presenting (202) a cross-sectional view (126) of the three-dimensional medical image on a user interface (124). The user interface includes a split input tool (128); the method further includes repeatedly: receiving (204) split edit data (132) from the split input tool, where the split input tool includes tool configuration parameters (130) configured to control how the split input tool generates the split edit data; collecting (206) tool use metadata (134), the tool use metadata description inputting the split edit data using the split input tool; modifying (208) the tool configuration parameters using the tool usage metadata; and construct (210) a segmentation (136) of the three-dimensional medical image using the segmentation editing data.

Description

Technical Field

[0001] This invention relates to medical imaging, and more particularly to the segmentation of medical images. Background Technology

[0002] Multiple imaging modalities exist that enable the acquisition of three-dimensional medical images describing the internal anatomy of an object. For example, magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), and single-photon emission tomography (SPECT) can be used to image the object. Three-dimensional medical images typically require segmentation. In segmentation, various normal and / or abnormal regions are identified and delineated. Segmentation is extremely useful for tracking tumor growth over time or for controlling the amount of radiation delivered to different organs in radiation therapy planning.

[0003] International patent application WO2017084871A1 discloses a system and a computer-implemented method for segmenting objects in medical images using a graphical segmentation interface. The graphical segmentation interface may include a set of segmentation tools to enable a user to obtain a first segmentation of an object in the image. The first segmentation can be represented by segmentation data. Interaction data can be obtained, indicating user interactions with a set of user interactions on the graphical segmentation interface, through which the first segmentation of the object is obtained. The system may include a processor configured to analyze the segmentation data and the interaction data to determine a set of optimized user interactions that, when performed by the user, yield a second segmentation similar to the first segmentation, but in a faster and more convenient manner. Videos can be generated for training the user by instructing them on the optimized set of user interactions. Summary of the Invention

[0004] The present invention provides a medical system, a computer program, and a method in the independent claims. Embodiments are given in the dependent claims.

[0005] In one aspect, the present invention provides a medical system including a memory storing machine-executable instructions. The medical system also includes a computing system. The execution of the machine-executable instructions causes the computing system to receive a three-dimensional medical image. The execution of the machine-executable instructions also causes the computing system to present a cross-sectional view of the three-dimensional medical image on a user interface. The user interface includes a segmentation input tool.

[0006] The execution of the machine-executable instructions also causes the computing system to repeatedly receive segmentation editing data from the segmentation input tool. The segmentation input tool includes tool configuration parameters configured to control how the segmentation input tool generates segmentation editing data.

[0007] The execution of the machine-executable instructions also causes the computing system to repeatedly collect tool usage metadata, which describes the segmentation input tool's input segmentation editing data. The execution of the machine-executable instructions also causes the computing system to repeatedly modify the tool configuration parameters using the tool usage metadata. Furthermore, the execution of the machine-executable instructions causes the computing system to repeatedly construct the segmentation results of the three-dimensional medical image using the segmentation editing data.

[0008] In another aspect, the present invention provides a computer program comprising machine-executable instructions for execution by a computing system. The execution of the machine-executable instructions causes the computing system to receive a three-dimensional medical image. The execution of the machine-executable instructions also causes the computing system to present a cross-sectional view of the three-dimensional medical image on a user interface. The user interface includes a segmentation input tool. The execution of the machine-executable instructions also causes the computing system to repeatedly receive segmentation editing data from the segmentation input tool.

[0009] The segmentation input tool includes tool configuration parameters configured to control how the segmentation input tool generates segmentation editing data. The execution of the machine-executable instructions also causes the computing system to repeatedly collect tool usage metadata describing the segmentation editing data input using the segmentation input tool. The execution of the machine-executable instructions also causes the computing system to repeatedly modify the tool configuration parameters using the tool usage metadata. Furthermore, the execution of the machine-executable instructions causes the computing system to repeatedly construct the segmentation results of the three-dimensional medical image using the segmentation editing data.

[0010] In another aspect, the present invention provides a method. The method includes receiving a three-dimensional medical image. The method also includes presenting a cross-sectional view of the three-dimensional medical image on a user interface. The user interface includes a segmentation input tool.

[0011] The method further includes repeatedly receiving segmentation editing data from the segmentation input tool. The segmentation input tool includes tool configuration parameters configured to control how the segmentation input tool generates the segmentation editing data. The method further includes repeatedly collecting tool usage metadata describing the input of the segmentation editing data using the segmentation input tool. The method further includes repeatedly modifying the tool configuration parameters using the tool usage metadata. The method further includes repeatedly constructing segmentation of the three-dimensional medical image using the segmentation editing data. Attached Figure Description

[0012] In the following description, preferred embodiments of the invention will be illustrated by way of example only and with reference to the accompanying drawings, in which:

[0013] Figure 1 An example of a medical system is illustrated.

[0014] Figure 2 The illustration shows the use of Figure 1 A flowchart of the methods used in medical systems.

[0015] Figure 3 This illustration shows another example of a medical system.

[0016] Figure 4 The illustration shows the use of Figure 3 A flowchart of the methods used in medical systems.

[0017] Figure 5 The illustration shows the construction of sub-image 500 from cross-sectional view 126.

[0018] Figure 6 An example of an image processing neural network 600 is illustrated.

[0019] Figure 7 The illustration shows an example of how the tool size of the segmentation input tool changes. List of reference numerals 100 Medical Systems 102 Computer 104 Computing System 106 Optional hardware interfaces 108 User Interface 110 Memory 120 Machine-executable instructions 122 Three-Dimensional Medical Images 124 Graphical User Interface 126 Cross-sectional View 128-segment input tool 130 Tool Configuration Parameters 132 Split and edit data 134 Tools use metadata 136 split 138. Presentation of segmentation 200 receives 3D medical images 202 Presents a cross-sectional view of a 3D medical image on the user interface. 204 Receive segmentation editing data from the segmentation input tool 206 The collection tool uses metadata, which describes the segmentation and editing data input using the segmentation input tool. 208. Use the tool to modify tool configuration parameters using metadata. 210 Using segmentation editing data to construct segmentation of 3D medical images Is editing complete for 212? 214 provides segmentation of 3D medical images after segmentation construction is completed. 300 Medical System 302 Magnetic Resonance Imaging System 304 magnet 306 Magnet Chamber 308 Imaging Area 309 Field of view 310 Magnetic Gradient Coil 312 Magnetic Gradient Coil Power Supply 314 RF coil 316 transceiver 318 Objects 320 Object Support 330 Pulse Sequence Command 332 k spatial data 400 Control medical imaging system to receive three-dimensional medical images 500 sub-images 600 Image Processing Neural Networks 602 Tool Size Prediction 700 Before size change 702 After dimensional changes 704 First Distance 706 Second Distance 708 contact point Detailed Implementation

[0020] In these figures, similarly numbered elements are equivalent elements or perform the same function. If the functions are equivalent, elements that have been discussed previously will not necessarily be discussed in later figures.

[0021] Examples can be helpful because they provide a user interface that can automatically adapt to the needs of the user, either by inputting segmentation editing data or controlling the segmentation input tool. Typically, switching between different tools and / or tool settings can be very tedious when manually segmenting 3D medical images. The behavior of the segmentation input tool varies depending on how it is used, and tool configuration parameters can be automatically changed by using tool metadata.

[0022] For example, a three-dimensional medical image can be a three-dimensional dataset, or it can be a stack of two-dimensional medical images.

[0023] In many medical imaging techniques, such as magnetic resonance imaging (MRI) or computed tomography (CT), two-dimensional thick slices representing the thickness of an object are repeatedly acquired. These can be used to create three-dimensional medical images.

[0024] The segmentation input tool configuration parameters can include data such as the segmentation input tool's attributes, and various properties such as the number or size of the affected segmentation regions. Various types of segmentation input tools can be specified, such as drawing tools for manually entering segmentation locations and drawing tools for modifying or adding segments. In some examples, a segment can be a set of lines or regions represented in different cross-sectional views of a 3D medical image. In other examples, a segment can be a grid specified in 3D space within a 3D medical image. For example, a segmentation input tool can be used to change, stretch, or deform this grid.

[0025] In another example, tool configuration parameters include tool size. The tool size used in this article may encompass the region or volume affected by the segmentation input tool. For example, a segmentation input tool can be used to draw or fill a region, and the tool size can indicate the size of the segmentation line or the region being drawn or filled. In other examples, the segmentation input tool can be a ball or other device used to push or shift a 3D mesh used to represent the segmentation. The size of the tool used to push or deform the mesh affects the speed and size of the edited region.

[0026] In another example, the memory also stores an edge detection algorithm. The execution of the machine-executable instructions further enables the computing system to use the edge detection algorithm to determine the edge density within a predetermined edge detection distance from the segmentation input tool in the cross-sectional view. The tool uses metadata to include the edge density. The execution of the machine-executable instructions also enables the computing system to adjust the tool size proportionally to changes in edge density as the segmentation input tool moves within the cross-sectional view.

[0027] For example, edge detection algorithms can be implemented in different ways. The algorithm can be implemented in a way that allows it to look for sudden changes in contrast or other image properties. In other examples, a neural network can be trained to identify edges within a predetermined distance around a segmentation input tool. Edge density can be used to represent, for example, the amount of fill in the detected edge at a predetermined edge detection distance from the segmentation input tool, or it can be used to determine the degree of complexity or folding that occurs within that distance. In either case, determining this edge density is relatively straightforward, and edge density can be a good indicator of how large the tool size should be. For example, if the edges have a very fine and detailed variation in quantity, using a smaller segmentation input tool may be more advantageous. However, if the edge density is small and it is essentially a straight line, then using a larger tool size may be beneficial, so that less human intervention is required to modify, smooth, or locate the segmentation.

[0028] In another example, the memory also stores an image processing neural network. The image processing neural network is configured to output a tool size prediction in response to receiving a sub-image selected from the cross-sectional view. For example, the sub-image could be an image centered on the location of the segmentation input tool. The execution of the machine-executable instructions further causes the computing system to generate the sub-image by selecting a region within the cross-sectional view located at a predetermined distance from the location of the segmentation input tool within the cross-sectional view. The execution of the machine-executable instructions further causes the computing system to receive the tool size prediction in response to inputting the sub-image into the image processing neural network. The execution of the machine-executable instructions further causes the computing system to change tool configuration parameters such that the tool size is set to the tool size prediction.

[0029] This embodiment may be beneficial because the tool size of the segmentation input tool will be automatically selected using an image processing neural network.

[0030] Image processing neural networks can be, for example, U-Net, ResNet, or other neural networks suitable for image processing, such as convolutional neural networks or neural networks with fully connected layers. For instance, an image processing neural network can be trained by selecting data and recording when a medical system operator manually changes the size of a tool. This can be used to generate training data. For example, the position of the image and segmentation input tool can be saved when the tool size changes. This can be used to collect data pairs that can be used to form training data for training the image processing neural network.

[0031] In another example, the tool uses metadata to include the segmentation tool speed. For example, this could be the speed of the segmentation input tool in a cross-sectional view. The execution of the machine-executable instructions also causes the computing system to: adjust the tool size such that changes in the tool size are proportional to changes in the segmentation tool speed, and / or automatically smooth the segmentation when the segmentation tool speed exceeds a predetermined threshold.

[0032] For a simple example, this can be achieved by measuring the speed of the mouse cursor or segmented input tool and using a rule-based algorithm that linearly increases the pen size as the mouse cursor speed increases, for example, pen size = (1 + step_function(velocity)). slope The velocity voxel is defined as follows: the step function ensures that the size is increased only after a certain mouse cursor speed threshold is exceeded.

[0033] This can be beneficial because as the tool moves faster, it may be necessary to modify larger portions of the segment simultaneously. Conversely, if the operator moves the segmentation input tool more slowly, they may want to make more or finer adjustments to the segmentation.

[0034] In another example, the execution of the machine-executable instructions also causes the computing system to use metadata to detect repetitive movements of the segmentation input tool. For example, if the segmentation input tool moves back and forth, or repeatedly moves to the same or adjacent voxels, this may indicate that the user is performing relative editing on the same or very close areas. The execution of the machine-executable instructions causes the computing system to increase the size of the segmentation input tool in response to detecting repetitive movements. For example, if the operator is using a spherical tool to distort the boundaries of a segment, increasing the tool size will reduce the need for repetitive movements. Automatically increasing the size of the segmentation input tool can reduce the amount of work or number of movements required for editing or inputting segments by the subject.

[0035] For example, repetitive motion, such as continuous "back and forth" movements, can be detected by determining the presence of repeating patterns. If detected, this will also cause the brush size to increase because it indicates that a larger, unstructured area is being drawn. These patterns can be detected, for example, via a Fourier transform of the velocity over time. Repetitive movements will then appear as frequency peaks.

[0036] In another example, the segmentation is a 3D mesh. The segmentation input tool is configured to perform any of the following: push the 3D mesh, distort the 3D mesh, or draw points in the 3D mesh, or a combination thereof.

[0037] In another example, the execution of the machine-executable instructions also causes the computing system to adjust the position of the segmentation input tool so that the tool remains in contact with the 3D mesh even when its size changes. For example, if the segmentation input tool is a sphere or other shape used to twist or push the 3D mesh, decreasing or increasing the tool size means that the tool's position relative to the mesh changes. Adjusting the tool to remain in contact with the 3D mesh reduces the likelihood of the mesh accidentally deforming in the wrong direction.

[0038] In another example, the tool configuration parameters include generating a distance perpendicular to the cross-sectional view targeted by the segmented editing data.

[0039] In another embodiment, the tool configuration parameters include the ball size.

[0040] In another embodiment, the tool configuration parameters include the active line diameter.

[0041] In another embodiment, the tool configuration parameters include point label size.

[0042] In another embodiment, the tool configuration parameters include the interactive region growth size.

[0043] In another example, the memory also stores a tool configuration neural network configured to output tool configuration parameter modifications in response to receiving tool usage metadata as input; the execution of the machine-executable instructions further causes the computing system to receive the tool configuration parameter modifications in response to inputting the tool usage metadata into the tool configuration neural network. The execution of the machine-executable instructions also causes the computing system to change the tool configuration parameters using the modifications. This embodiment may be advantageous because it can provide a method for automatically configuring tool configuration parameters. The tool configuration neural network can be, for example, a U-net, ResNet, a convolutional network, or a neural network with fully connected layers. The tool usage metadata can be digitally encoded before being input into the tool configuration network. Training data can be obtained by storing the tool usage metadata during use and then recording when the operator manually changes the tool configuration parameters. These data pairs can be saved and then used to train the tool configuration neural network.

[0044] One approach to training a neural network is to record specific user preferences by storing a set of pen parameters selected by the user at time t0, along with mouse movements from time t0 to t1, until the user changes the pen parameters again. By pairing the mouse movements recorded between t0 and t1 with the user's selection at t0, a model can be trained to predict said set of pen parameters based on mouse movements.

[0045] In another example, the execution of the machine-executable instructions also causes the computing system to record manual tool configuration parameter modifications by the medical system operator, as well as tool usage metadata. The execution of the machine-executable instructions causes the computing system to use the manual tool configuration parameter modifications and tool usage metadata to train a tool configuration neural network, thereby providing a user-specific tool configuration neural network. This embodiment can be advantageous because the system can be automatically configured or adjusted to suit the preferences of a particular user.

[0046] In another example, the medical imaging system further includes a medical imaging system configured to image at least a portion of an object. The execution of the machine-executable instructions also causes the computing system to control the medical imaging system to acquire the three-dimensional medical images. The acquired three-dimensional medical images can then be used in the steps described above to construct segmentation.

[0047] In another example, the three-dimensional medical image is a magnetic resonance image.

[0048] In another embodiment, the three-dimensional medical image is a computed tomography (CT) image.

[0049] In another embodiment, the three-dimensional medical image is a three-dimensional ultrasound image.

[0050] In another embodiment, the three-dimensional medical image is a positron emission tomography (PET) image.

[0051] In another embodiment, the three-dimensional medical image is a single-photon emission computed tomography (SPECT) image.

[0052] In another example, the execution of the machine-executable instructions also enables the computing system to provide segmentation of a three-dimensional medical image after the segmentation is constructed.

[0053] Figure 1 An example of a medical system 100 is illustrated. Figure 1 The medical system 100 depicted is shown as including a computer 102. Computer 102 is intended to represent one or more computers located at one or more locations. Computer 102 is also shown as including a computing system 104. Computing system 104 is intended to represent one or more computing systems or computing cores located at one or more locations. Computing system 104 is connected to an optional hardware interface 106. If the medical system 100 includes additional components, computing system 104 can exchange data with and / or control those components using hardware interface 106. Computing system 104 is also shown as communicating with a user interface 108 and memory 110. Memory 110 is intended to represent one or more memories that are communicating with or accessible to computing system 104.

[0054] Memory 110 is shown storing machine-executable instructions 120. The machine-executable instructions 120 enable the computing system 104 to perform various control, computation, and data manipulation tasks. Memory 110 is also shown containing three-dimensional medical images 122.

[0055] User interface 108 is shown as including graphical user interface 124. Graphical user interface 124 shows a cross-sectional view 126 of a three-dimensional medical image 122. Graphical user interface 124 has a segmentation input tool 128 for inputting segmentation editing data 132 to provide a segmentation 136 of the three-dimensional medical image 122. Segmentation 136 is shown as being stored in memory 110, and portions 138 of segmentation 136 are presented in the graphical user interface 124. The operator can use the segmentation input tool 128 to manipulate and further edit the segmentation 136.

[0056] The memory 110 is also shown to contain tool configuration parameters 130. Tool configuration parameters 130 control the behavior and other attributes of the segmentation input tool 128. When using the segmentation input tool 128, the operator can input segmentation editing data 132 for constructing and / or modifying segments 136. During the use of the segmentation input tool 128, tool usage metadata 134 is generated. This can provide information such as the speed at which the segmentation input tool 128 moves in the cross-sectional view 126, as well as other information such as image attributes or other configuration parameters of the cross-sectional view 126.

[0057] Figure 2 The illustrated operation is shown. Figure 1 A flowchart of a method for a medical system 100. In step 200, a three-dimensional medical system 122 is received. In step 202, a cross-sectional view 126 of the three-dimensional medical system 122 is presented on a user interface 108; in this case, a graphical user interface 124. In step 204, the operator manipulates a segmentation input tool 128 to generate segmentation editing data 132. In step 206, when the segmentation editing data 132 is generated, the tool uses metadata 134. In step 208, the tool uses the metadata 134 to modify tool configuration parameters 130. This can be achieved through an algorithm or neural network, as described below. In step 210, a segment 136 is constructed or edited using the segmentation editing data 132. Step 212 is a question box asking: Is the editing of the segment complete? If the answer is no, the method returns to step 204, where steps 204, 206, 208, and 210 are executed in a loop until the editing of segment 136 is complete. The exact order of steps 204, 206, 208, and 210 is not fixed, and not all of these steps need to be performed in every loop.

[0058] In block 212, if editing is complete, the method continues to block 214, where a segmentation 136 of the three-dimensional medical system 122 is provided. In some instances, the segmentation may be stored together with the three-dimensional medical image 122. In some examples, the segmentation 136 may be displayed overlaid on the three-dimensional medical system 122. In other cases, the segmentation 136 may be provided for other uses, such as radiation therapy planning.

[0059] Figure 3 Another example of a medical system 300 is illustrated. Figure 3 In this context, the medical system 300 also includes a magnetic resonance imaging system 302. Generally, the magnetic resonance imaging system 302 is intended to represent a medical imaging system. For example, the magnetic resonance imaging system 302 can be replaced by a computed tomography system, a three-dimensional ultrasound system, a single-photon emission tomography system, or a positron emission tomography system.

[0060] like Figure 3 The illustrated medical system 300 also includes a magnetic resonance imaging (MRI) system 302. The MRI system 302 includes a magnet 304. The magnet 304 is a superconducting cylindrical magnet with a bore 306 passing through it. Different types of magnets are also possible; for example, split cylindrical magnets and so-called open magnets can also be used. A split cylindrical magnet is similar to a standard cylindrical magnet, except that the cryostat has been split into two parts to allow access to the isoplanar surface of the magnet; such a magnet can be used, for example, in conjunction with charged particle beam therapy. An open magnet has two magnet sections, one on top of the other, with a space large enough between them to accommodate the object: the arrangement of the two sections is similar to that of Helmholtz coils. Open magnets are popular because the object is less restricted. An assembly of superconducting coils is located inside the cryostat of the cylindrical magnet.

[0061] Within the bore 306 of the cylindrical magnet 304, there exists an imaging region 308 in which the magnetic field is sufficiently strong and uniform to perform magnetic resonance imaging. A field of view 309 within the imaging region 308 is shown. Typically, k-space data is acquired for the field of view 309. A region of interest may be identical to or a subvolume of the field of view 309. An object 318 is shown supported by an object support 320 such that at least a portion of the object 318 is within both the imaging region 308 and the field of view 309.

[0062] The magnet's bore 306 also contains an assembly of magnetic field gradient coils 310, which are used to acquire primary k-space data for spatial encoding of magnetic spins within the imaging region 308 of the magnet 304. The magnetic field gradient coils 310 are connected to a magnetic field gradient coil power supply 312. The magnetic field gradient coils 310 are intended to be representative. Typically, the magnetic field gradient coils 310 comprise an assembly of three separate coils for spatial encoding in three orthogonal spatial directions. The magnetic field gradient power supply provides current to the magnetic field gradient coils. The current supplied to the magnetic field gradient coils 310 is time-controlled and can be either slanted or pulsed.

[0063] Adjacent to the imaging region 308 is an RF coil 314, which is used to manipulate the orientation of the magnetic spins within the imaging region 308 and to receive RF transmissions from spins also located within the imaging region 308. The RF antenna may comprise multiple coil elements. The RF antenna may also be referred to as a channel or antenna. The RF coil 314 is connected to an RF transceiver 316. The RF coil 314 and the RF transceiver 316 may be replaced by separate transmit and receive coils, and separate transmitters and receivers. It should be understood that the RF coil 314 and the RF transceiver 316 are representative. The RF coil 314 is intended to also represent a dedicated transmit antenna and a dedicated receive antenna. Similarly, the transceiver 316 may also represent separate transmitters and receivers. The RF coil 314 may also have multiple receive / transmit elements, and the RF transceiver 316 may have multiple receive / transmit channels.

[0064] Transceiver 316 and gradient controller 312 are shown as hardware interface 106 connected to computer system 102.

[0065] Memory 110 is again shown as containing machine-executable instructions 120 and an image processing neural network 122. Memory 110 is also shown as containing pulse sequence commands 330. Pulse sequence commands 330 are commands or data that can be converted into commands, used to control the magnetic resonance imaging system 302 to acquire k-space data 332 according to a three-dimensional medical magnetic resonance imaging protocol. Memory 110 is shown as containing k-space data 332, which is acquired by controlling the magnetic resonance imaging system 302 using pulse sequence commands 330. In this example, the three-dimensional medical image 122 is reconstructed from the k-space data 332.

[0066] Figure 4 The illustrated operation is shown. Figure 3The flowchart illustrates the method for using the medical system 300. In block 400, the medical imaging system (in this example, the magnetic resonance imaging system 302) is controlled to receive three-dimensional medical images 122. This can be achieved, for example, by controlling the magnetic resonance imaging system 302 via pulse sequence commands 330. This causes the magnetic resonance imaging system 302 to acquire k-space data 332. The k-space data 332 can be reconstructed into a magnetic resonance image, or, in this case, into the three-dimensional medical system 122.

[0067] Figure 5 The illustration shows the construction of sub-image 500 from cross-sectional view 126. Figure 5 A portion of the graphical user interface 124 is illustrated. A box 500 surrounds the segmentation input tool 128 and is oriented around it. This box 500 is the region used to select a sub-image 500 from the cross-sectional view 126. Since the sub-image 500 is centered on the segmentation input tool 128, the image in the sub-image 500 describes the region that the segmentation input tool 128 is currently editing. For example, the sub-image 500 can be inserted into an edge detection algorithm to determine the edge density within a predetermined edge detection distance from the segmentation input tool. In other examples, the sub-image can be inserted into an image processing neural network.

[0068] Figure 6 An example of an image processing neural network 600 is illustrated. The image processing neural network 600 is configured to receive a sub-image 500 as input. It is then trained to output an tool size prediction 602. For example, the tool size prediction 602 can be encoded by the output of the image processing neural network 600. As the segmentation input tool 128 moves in the cross-sectional view 126, the sub-images 500 surrounding the segmentation input tool 128 can be continuously or repeatedly input into the image processing neural network 600, thereby providing the tool size prediction 602 in a more or less continuous manner. This allows for continuous or repetitive adjustment of the tool size of the segmentation input tool.

[0069] When the tool size changes, this can affect the editing of segment 136. For example, if the tool size decreases, the tool will no longer be in contact with segment 136. For example, if the tool size increases, it may cause the segment to be distorted in an undesirable way.

[0070] Figure 7The diagram illustrates how to reduce or eliminate these problems. Two views are shown: one before the size change, and a second view 702 showing the size after the change. A cross-sectional view 126 of the three-dimensional medical system 122 is shown in both views. The presentation of the segment 138 is also shown. In the first view 700, the segment input tool 128 is larger and is shown at a first distance 704 from the presentation of the segment 138. In the second view 702, the size of the segment input tool 128 is reduced. However, the segment input tool 128 automatically moves closer to the presentation of the segment 138. The center of the segment input tool 128 in the second view 702 is shown at a second distance 706, which is smaller than the first distance 704 in the first view 700. In views 700 and 702, the segment input tool 128 maintains the same contact point 708 before and after the size change.

[0071] It should be understood that one or more of the foregoing examples or embodiments of the present invention may be combined, as long as the combined embodiments are not mutually exclusive.

[0072] As those skilled in the art will recognize, several aspects of the invention can be implemented as apparatus, method, or computer program product. Therefore, aspects of the invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, which can be collectively referred to herein as “circuit,” “module,” or “system.” Furthermore, aspects of the invention can take the form of computer program products implemented in one or more computer-readable media having computer-executable code implemented thereon.

[0073] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. As used herein, "computer-readable storage medium" includes any tangible storage medium that can store instructions executable by a processor or computing system of a computing device. The computer-readable storage medium may be referred to as a "computer-readable non-transient storage medium." The computer-readable storage medium may also be referred to as a tangible computer-readable medium. In some embodiments, the computer-readable storage medium may also be able to store data accessible by the computing system of the computing device. Examples of computer-readable storage media include, but are not limited to: floppy disks, magnetic hard disk drives, solid-state drives, flash memory, USB thumb drives, random access memory (RAM), read-only memory (ROM), optical disks, magneto-optical disks, and register files of computing systems. Examples of optical disks include compact optical disks (CDs) and digital multi-purpose optical disks (DVDs), such as CD-ROMs, CD-RWs, CD-Rs, DVD-ROMs, DVD-RWs, or DVD-R discs. The term computer-readable storage medium also refers to various types of recording media accessible by the computer device via a network or communication link. For example, data can be retrieved via a modem, via the Internet, or via a local area network. Computer-executable code embodied on a computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, or any suitable combination of the foregoing.

[0074] Computer-readable signal media may include propagated data signals having computer-executable code implemented therein, for example, in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and is capable of transmitting, propagating, or conveying a program for use by or in connection with an instruction execution system, apparatus, or device.

[0075] "Computer memory" or "memory" is an example of a computer-readable storage medium. Computer memory is any memory that can be directly accessed by a computing system. "Computer storage device" or "storage device" is another example of a computer-readable storage medium. A computer storage device is any non-volatile computer-readable storage medium. In some embodiments, a computer storage device may also be computer memory, or vice versa.

[0076] As used herein, "computing system" encompasses electronic components capable of executing programs or machine-executable instructions or computer-executable code. References to computing systems that include examples of "computing systems" should be interpreted as potentially including more than one computing system or processing core. A computing system can, for example, be a multi-core processor. A computing system can also refer to a collection of computing systems within a single computer system or distributed across multiple computer systems. The term computing system should also be interpreted as potentially referring to a collection or network of computing devices, each including a processor or multiple computing systems. Machine-executable code or instructions can be executed by multiple computing systems or processors, which may be within the same computing device or even distributed across multiple computing devices.

[0077] Machine-executable instructions or computer-executable code may include instructions or programs that cause a processor or other computing system to perform one aspect of the invention. Computer-executable code for performing operations targeting the aspects of the invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages, and compiled into machine-executable instructions. In some cases, the computer-executable code may be used in the form of a high-level language or in a pre-compiled form in conjunction with an interpreter that generates machine-executable instructions on the fly. In other cases, the machine-executable instructions or computer-executable code may be in the form of programming against a programmable gate array.

[0078] The computer-executable code can run as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or can establish a connection with an external computer (e.g., via the Internet using an Internet service provider).

[0079] Various aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block or portion of a block in a flowchart, illustration, and / or block diagram can be implemented, where applicable, by computer program instructions in the form of computer-executable code. It should also be understood that combinations of blocks in different flowcharts, illustrations, and / or block diagrams can be combined when not mutually exclusive. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that instructions executable via the computer's memory or other programmable data processing apparatus create units for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0080] These machine-executable instructions or computer program instructions may also be stored in a computer-readable medium that is capable of directing a computer, other programmable data processing apparatus or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing comprising instructions that implement the functions / actions specified in flowcharts and / or one or more block diagrams.

[0081] The machine-executable instructions or computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions running on the computer or other programmable apparatus provide for implementing the functions / actions specified in the flowchart and / or one or more block diagram boxes.

[0082] As used herein, a “user interface” is an interface that allows a user or operator to interact with a computer or computer system. A “user interface” can also be referred to as a “human-machine interface device.” A user interface can provide information or data to an operator and / or receive information or data from an operator. A user interface enables input from an operator to be received by the computer and can provide output from the computer to the user. In other words, a user interface allows an operator to control or manipulate a computer, and the interface allows the computer to indicate the effects of the operator’s control or manipulation. The display of data or information on a monitor or graphical user interface is an example of providing information to an operator. Receiving data via a keyboard, mouse, trackball, touchpad, pointing stick, graphics tablet, joystick, gamepad, webcam, head-mounted device, pedal, wired gloves, remote control, and accelerometer are all examples of user interface components that implement the receiving of information or data from an operator.

[0083] As used herein, "hardware interface" encompasses an interface that enables a computer system to interact with and / or control external computing devices and / or apparatuses. A hardware interface allows the computing system to send control signals or instructions to external computing devices and / or apparatuses. A hardware interface can also enable the computing system to exchange data with external computing devices and / or apparatuses. Examples of hardware interfaces include, but are not limited to: Universal Serial Bus (USB), IEEE 1394 port, parallel port, IEEE 1284 port, serial port, RS232 port, IEEE488 port, Bluetooth connectivity, wireless LAN connectivity, TCP / IP connectivity, Ethernet connectivity, control voltage interface, MIDI interface, analog input interface, and digital input interface.

[0084] As used herein, "display" or "display device" encompasses an output device or user interface suitable for displaying images or data. A display may output visual, audio, and / or tactile data. Examples of displays include, but are not limited to: computer monitors, television screens, touchscreens, tactile electronic displays, Braille screens, etc.

[0085] Cathode ray tubes (CRTs), storage tubes, bistable displays, electronic paper, vector displays, flat panel displays, vacuum fluorescent displays (VFs), light-emitting diode (LED) displays, electroluminescent displays (ELDs), plasma display panels (PDPs), liquid crystal displays (LCDs), organic light-emitting diode (OLED) displays, projectors, and head-mounted displays.

[0086] Although the invention has been illustrated and described in detail in the accompanying drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary, and not restrictive. The invention is not limited to the disclosed embodiments.

[0087] Those skilled in the art, through studying the accompanying drawings, disclosure, and claims, will understand and implement other variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single processor or other unit can perform the functions of several items recited in the claims. Although specific measures are recited in dissimilar dependent claims, this does not imply that combinations of these measures cannot be advantageously used. Computer programs can be stored / distributed on suitable media such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems. Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A medical system (100, 300) comprising: a memory (110) storing machine executable instructions (120); a computing system (104), wherein execution of the machine executable instructions result in the computing system: receiving (200) a three-dimensional medical image (122); presenting (202) a cross-sectional view (126) of the three-dimensional medical image on a user interface (124), wherein the user interface comprises a segmentation input tool (128); wherein execution of the machine executable instructions further result in the computing system repeatedly: receiving (204) segmentation edit data (132) from the segmentation input tool, wherein the segmentation input tool comprises tool configuration parameters (130) configured to control how the segmentation input tool generates the segmentation edit data; collecting (206) tool usage metadata (134) describing use of the segmentation input tool to input the segmentation edit data; modifying (208) the tool configuration parameters (130) using the tool usage metadata; and constructing (210) a segmentation (136) of the three-dimensional medical image using the segmentation edit data.

2. The medical system of claim 1, wherein, the tool configuration parameters comprise a tool size.

3. The medical system of claim 2, wherein, the memory further stores an edge detection algorithm, wherein execution of the machine executable instructions further result in the computing system: determining, using the edge detection algorithm, an edge density within a predetermined edge detection distance from the segmentation input tool in the cross-sectional view, wherein the tool usage metadata comprises the edge density; and adjusting the tool size in proportion to changes in the edge density as the segmentation input tool moves in the cross-sectional view.

4. The medical system of claim 2, wherein, the memory further stores an image processing neural network (600), wherein the image processing neural network is configured to output a tool size prediction in response to receiving a sub-image (500) selected from the cross-sectional view, wherein execution of the machine executable instructions further result in the computing system: generating the sub-image by selecting a region of the cross-sectional view positioned within a predetermined distance from a location of the segmentation input tool within the cross-sectional view; receiving the tool size prediction in response to inputting the sub-image into the image processing neural network; and modifying the tool configuration parameters such that the tool size is set to the tool size prediction.

5. The medical system of claim 2, 3, or 4, wherein, the tool usage metadata comprises a segmentation tool speed, wherein execution of the machine executable instructions further result in the computing system: adjusting the tool size such that changes in tool size are proportional to changes in the segmentation tool speed, and / or automatically smoothing the segmentation when the segmentation tool speed exceeds a predetermined threshold.

6. The medical system of any of the preceding claims, wherein execution of the machine executable instructions further result in the computing system: detecting repetitive motion of the segmentation input tool using the tool usage metadata; and In response to the detected repetitive movement of the segmentation input tool, the size of the segmentation input tool is increased.

7. The medical system of any of claims 2 to 6, wherein, The segmentation is divided into a three-dimensional mesh, wherein the segmentation input tool is configured to perform any of the following: pushing the three-dimensional mesh, distorting the three-dimensional mesh, drawing points in the three-dimensional mesh, and combinations thereof.

8. The medical system of claim 7, wherein, The execution of the machine-executable instructions also causes the computing system to adjust the position of the segmentation input tool so that the segmentation input tool remains in contact with the three-dimensional mesh when the tool size changes (708).

9. The medical system of any one of the preceding claims, wherein, The tool configuration parameters include any of the following: distance perpendicular to the cross-sectional view for which the segmented editing data is generated, tool type, sphere size, live line diameter, point annotation size, interactive region growth size, and combinations thereof.

10. The medical system of any one of the preceding claims, wherein, The memory also stores a tool configuration neural network configured to output tool configuration parameter modifications in response to receiving metadata as input from the tool; wherein, the execution of the machine-executable instructions further enables the computing system to: In response to receiving a modification of the tool configuration parameters by inputting the tool with metadata into the tool configuration neural network; and Use the tool configuration parameter modification tool to modify the tool configuration parameters.

11. The medical system of claim 10, wherein, The execution of the machine-executable instructions also enables the computing system to: Record the operator's manual tool configuration parameter modifications and tool usage metadata of the medical system; and The manual tool configuration parameters are modified, and the tool uses metadata to train the tool configuration neural network to provide a user-specific tool configuration neural network.

12. The medical system of any one of the preceding claims, wherein, The medical imaging system further includes a medical imaging system (302) configured to image at least a portion of an object, wherein the execution of the machine-executable instructions further causes the computing system to control the medical imaging system to acquire the three-dimensional medical images.

13. The medical system of any one of the preceding claims, wherein, The three-dimensional medical image is any of the following: magnetic resonance imaging, computed tomography (CT) images, three-dimensional ultrasound images, positron emission tomography (PET) images, and single-photon emission tomography (SPECT) images.

14. A computer program comprising machine executable instructions (120) for execution by a computing system (120), wherein, The execution of the machine-executable instructions enables the computing system to: Receive (200) three-dimensional medical images (122); A cross-sectional view (126) of the three-dimensional medical image is presented (202) on a user interface (124), wherein the user interface includes a segmentation input tool (128). The execution of the machine-executable instructions also causes the computing system to repeatedly: Receive (204) segmentation editing data (132) from the segmentation input tool, wherein the segmentation input tool includes tool configuration parameters (130) configured to control how the segmentation input tool generates the segmentation editing data; The collection (206) tool uses metadata (134) to describe the segmentation editing data input using the segmentation input tool; Use the tool to modify (208) the tool configuration parameters (130) using metadata; and The segmentation editing data is used to construct (210) a segmentation (136) of the three-dimensional medical image.

15. A method wherein, The method includes: Receive (200) three-dimensional medical images (122); A cross-sectional view (126) of the three-dimensional medical image is presented (202) on a user interface (124), wherein the user interface includes a segmentation input tool (128). The execution of the machine-executable instructions also causes the computing system to repeatedly: Receive (204) segmentation editing data (132) from the segmentation input tool, wherein the segmentation input tool includes tool configuration parameters (130) configured to control how the segmentation input tool generates the segmentation editing data; The collection (206) tool uses metadata (134) to describe the segmentation editing data input using the segmentation input tool; The tool is used to modify (208) the tool configuration parameters using metadata; and The segmentation editing data is used to construct (210) a segmentation (136) of the three-dimensional medical image.

Images