3D image segmentation

Abstract

A computer-implemented method includes: receiving a 3D image comprising objects depicted in the image, the 3D image comprising an ordered set of 2D images; determining contours surrounding objects in a first 2D image of the 2D images; and determining contours surrounding objects in a second 2D image of the 2D images, the second 2D image being non-adjacent to the first 2D image in the ordered set, the ordered set having an intermediate region between the first and second 2D images comprising one or more intermediate 2D images. In each of the first and second 2D images, the interior of the contour is classified as foreground, and the exterior of the contour is classified as background. The method further includes performing a 3D geodesic distance calculation to classify points in the intermediate region as either background or foreground.

Description

Background Technology

[0001] Image segmentation involves assigning labels to elements, classifying them as either "foreground" (elements to be labeled) or "background" (elements to be excluded from the labels). An example would be outlining all the cars in an image of a street. The cars would be labeled "foreground," and the street, signs, and houses would be labeled "background." Another example would be segmenting medical scans, such as outlining organs or tumors as foreground elements in an image.

[0002] An example of automated image segmentation is disclosed in US10,235,605. This uses the concept of "geodesic distance" between points in an image. Geodesic distance is a distance metric that takes into account one or more parameters in an image, such as pixel intensity. Intensity can be thought of as similar to altitude on a map: the shortest distance between two points might be going around a mountain rather than crossing it. In other words, the distance metric is weighted by the intensity (or other such parameter) of pixels or points along the path. Pixels or voxels in a 3D image can be passed to a geodesic distance algorithm to compute the resulting foreground segmentation. Summary of the Invention

[0003] According to one aspect disclosed herein, a computer-implemented method is provided, comprising: receiving a 3D image including objects depicted in the image, the 3D image comprising an ordered set of 2D images; determining contours surrounding objects in a first 2D image of the 2D images; and determining contours surrounding objects in a second 2D image of the 2D images. The second 2D image is not adjacent to the first image in the ordered set, and has an intermediate region within the ordered set between the first and second 2D images comprising one or more of the 2D images. In each of the first and second 2D images, the interior of the contour is classified as foreground, and the exterior of the contour is classified as background. The method further comprises performing a 3D geodesic distance calculation to classify points in the intermediate region as foreground or background.

[0004] The 3D geodesic distance calculation includes, for each of a plurality of points in the intermediate region: calculating the shortest foreground distance and calculating the shortest background distance, wherein the shortest foreground distance is the shortest 3D geodesic distance from the point to the foreground and the shortest background distance is the shortest 3D geodesic distance from the point to the background; and classifying the point as foreground if the shortest foreground distance is shorter than the shortest background distance, but classifying the point as background if the shortest background distance is shorter than the shortest foreground distance.

[0005] This synopsis is provided to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The claimed subject matter is also not limited to implementations that address any or all of the shortcomings mentioned herein. Attached Figure Description

[0006] To aid in understanding embodiments of this disclosure and to illustrate how such embodiments can be implemented, reference is made to the accompanying drawings by way of example only, wherein:

[0007] Figure 1 This is a schematic block diagram of a system for image segmentation.

[0008] Figure 2 This is a flowchart of a method for segmenting 2D images.

[0009] Figures 3A to 3E An example process for segmenting 2D image slices is shown.

[0010] Figures 4 to 9 An example tool for refining image segmentation is shown.

[0011] Figure 10 This is a flowchart of a method for segmenting 3D images.

[0012] Figure 11 This is a schematic diagram of 3D interpolation between 2D image slices.

[0013] Figures 12A-12B An example of 3D segmentation is shown.

[0014] Figure 13 The illustration schematically shows the independent segmentation of different 3D regions, and

[0015] Figure 14 The user interface options for switching between interpolation modes are shown. Detailed Implementation

[0016] The following describes systems and methods for performing 2D and 3D calculations of geodesic features to segment 3D images, such as user-driven image labeling for 3D images. A 3D image is an image that comprises an ordered collection of 2D images; in the case of a static 3D image, it comprises stacked 2D image “slices” across a volume. For example, an example of 3D labeling would be segmenting tissue in a 3D medical image scan, such as outlining the bladder in a CT (computed tomography) image.

[0017] The desired outcome is to allow 3D image segmentation without requiring the user to manually segment or refine the segmentation of each individual 2D image slice in the stack, while still providing results that are broadly consistent with the user's expectations. To achieve this, a geodesic distance algorithm with a novel approach that combines 2D and 3D computation is utilized below.

[0018] In this embodiment, the algorithm will only calculate the geodesic distance between "locked" 2D slices. The 2D geodesic distance calculation is used to compute each "locked" 2D slice, and then the 3D computation is performed as a single 3D computation only on the unlocked 2D slices. Since most 3D images are labeled in 2D on a computer screen, it is best to implement this algorithm in a way that avoids inconsistent or unintended behavior in 2D slices "off-screen".

[0019] As an optional optimization, if the user modifies one of the 2D slices, the algorithm does not need to perform a full 3D calculation for each modification of the "foreground" and "background" labels.

[0020] In the algorithm's operation, the user can start with any 2D slice of a 3D image. He or she first marks foreground or background regions within the 2D image slice. Once this is complete, the geodesic distance algorithm is used to segment the proposed foreground regions only on the 2D slices the user has already interacted with. At this stage, no 3D calculations are performed on the rest of the image stack. This results in the complete 2D segmentation the user expects within a single slice. See, for example... Figure 12A .

[0021] The user will then navigate to another 2D slice that is adjacent to or several slices away from the first 2D slice. The same user interaction is repeated, marking the foreground and / or background regions to complete the 2D segmentation of the currently interacting slice. The result of this segmentation should be presented to the user immediately and should also match the user's expectations for 2D segmentation.

[0022] Now that interaction with two 2D slices has occurred, automated 3D geodesic distance calculation is performed between the labeled slices. In this embodiment, this is done by converting the interacting 2D slices into brush maps of only foreground and background. If a pixel is within any of the 2D segments, it is labeled as foreground; otherwise, all other pixels on the interacting 2D slice are labeled as background. This provides the geodesic distance algorithm with a strong signal of the area the user has already interacted with. This method also effectively “locks” the interacting slices in place. The geodesic distance algorithm attempts to find segments on unlabeled voxels. However, since all voxels have already been labeled on the 2D slices, the resulting segments will not change.

[0023] Then, the user has at least two additional options. One is to interact with a new, unsegmented 2D slice above or below the currently interacting slice. Alternatively, in another option, the user can choose to modify the 3D segmentation on the uninteracted slice. If the user chooses to interact with a new slice, the above process is repeated, and 3D calculations are performed between the closest interacting 2D slice and the new 2D slice. See, for example. Figure 12B .

[0024] As an optional optimization, the algorithm does not calculate the full 3D geodesic distance between the topmost and bottommost slices, as this would produce the same result. Instead, it only needs to recalculate between the last interacting slice and the nearest "locked" 2D slice.

[0025] If a 2D slice is modified between two interacting 2D slices, 2D computations are still performed on the interacting slice. However, two 3D computations will be performed between that slice and the interacting slices above and below it. In embodiments running on multi-core computers, the two 3D computations can be run in parallel to eliminate the cost of running two 3D computations.

[0026] In embodiments, this use of the geodesic distance algorithm can be leveraged to provide any one or more of the following advantages or similar advantages: I) Users do not need to view “locked” slices because the algorithm guarantees they will not change, thus saving users time; II) Expected results of 3D segmentation based on previous user interactions; III) Excellent computational speedup and efficiency resulting from initially performing calculations only in 2D (allowing users immediate feedback on the results of their interactions); IV) 3D computation is accelerated and becomes more efficient by calculating only the 3D segmentation between “locked” 2D slices.

[0027] Figure 1 A block diagram of an example system for implementing the techniques disclosed herein is provided. The system includes an image capture device 102, a processing unit 104, a memory 106, a user input device 108, and a display device 110. Each of the image capture device 102, memory 106, user input device 108, and display device 110 is operatively coupled to the processing unit 104.

[0028] Image capture device 102 may take the form of, for example, a still image camera, video camera, or medical scanner (such as an X-ray camera, magnetic resonance imaging (MRI) scanner, computed tomography (CT) scanner, positron emission tomography (PET) scanner, ultrasound scanner, etc.). It is configured to capture 3D images, which may be still 3D images comprising stacked 2D images through a body, or video images comprising a series of 2D frames. Regardless of the form the image takes, image capture device 102 is arranged to provide the captured image to processing device 104. The image may be stored in a region of memory 106 and accessed by processing device 104 via memory 106. It should be understood that... Figure 1 The arrangement shown is slightly illustrative.

[0029] Processing device 104 includes one or more processing units implemented in one or more dies, IC (integrated circuit) packages, and / or housings located in one or more geographical locations. Memory 106 includes one or more memory cells implemented in one or more memory media in one or more housings located in one or more geographical locations. The memory cells may or may not be integrated with the processing units in the same housing. Distributed storage and / or computing techniques are known in the art when needed. For example, multiple processing units and / or memory cells may be networked together via networks such as local area networks (e.g., LANs within buildings such as hospitals) and / or wide area networks or interconnected networks such as the Internet. Alternatively, processing device 104 and / or memory 106 may all be integrated into the same housing of the same computer terminal.

[0030] Regardless of the approach, each of the one or more processing units can take any suitable form known in the art, such as a general-purpose central processing unit (CPU) or a dedicated form of coprocessor or accelerator processor (such as a graphics processing unit (GPU), digital signal processor (DSP), etc.). Each of the one or more processing units can include one or more cores. Each of the one or more memory units can take any suitable storage medium known in the art, such as: magnetic storage media, such as hard disk drives, magnetic tape drives, etc.; or electronic storage media, such as solid-state drives (SSDs), flash memory, or EEPROMs; or optical storage media, such as optical disc drives or glass- or memory crystal-based storage devices, etc. When it is mentioned herein that some data items are stored in memory 106 or its areas, this can mean that they are stored in any part of any one or more memory devices constituting memory 106. When it is mentioned that software runs on a processing device, this means that it can run on any one or more processing units constituting processing device 104.

[0031] User input device 108 may include any one or more suitable input devices known in the art for receiving input from a user. In embodiments, it includes at least one pointing device, such as a mouse, stylus, touchscreen, touchpad, and / or trackball, that enables a user to draw in an image rendered on display 110. Alternatively or additionally, user input device 108 may include one or more other input devices, such as a keyboard or voice control unit. The reference herein to receiving input from a user via user input device 108 may mean via any one or more user input devices constituting user input device 108. The user input devices may or may not be incorporated in the same housing as the processing units of processing device 104.

[0032] Display device 110 can take any suitable form to output images, such as an LED screen, LCD (liquid crystal display), plasma screen, electronic ink display, or conventional cathode ray tube. Optionally, in an embodiment, it can take the form of a touchscreen, thus also forming at least part of user input device 108. Input can be achieved via touch by a user's finger and / or using a stylus. Display device 110 may or may not be integrated into the same housing as the processing units(s) of processing device 104.

[0033] The system also includes image processing software 112, which includes a rendering module 114, a UI (user interface) module 116, a segmentation module 118, and a refinement module 120. The image processing software 112, including each of these modules 114, 116, 118, and 120, is implemented as software code stored in memory 106 and arranged to run on processing device 104. The code modules 114, 116, 118, and 120 are configured to perform various operations, respectively, during runtime, related to: rendering an image on display 110, enabling the user to interact with the image via one or more UI tools (such as brush tools), segmenting the image, and refining the segmentation.

[0034] Figure 2A flowchart of a method for determining the contours surrounding an object in a 2D image (such as a 2D slice through a body) is provided. The method begins at step 202, where a 2D image is captured via image capture device 102. The captured image is provided to processing device 104 for processing. This may include storing the captured image in memory 106, and the processing may include processing the stored image from memory 106. In step 204, rendering module 114, running on processing device 104, renders the captured image to a user via display device 110. In step 206, segmentation module 118 performs initial segmentation, dividing the image into at least one foreground object and a background region. Segmentation includes determining the contours surrounding the object, which will be discussed in more detail later. For the purposes of this application, “contour” means the outline surrounding the entire perimeter of the object. Segmentation may be performed in a fully automatic or partially automatic manner based on user input received from the user, or in a fully manual manner based on user input from the user. Any such user input is received via UI module 116 through user input device 108. In step 208, the rendering module 114 outputs the contour determined by the segmentation module 118 through the display 110.

[0035] In step 210, UI module 116 receives user input from the user via user input device 108 to refine the contour initially determined by segmentation module 118. According to this disclosure, this includes the user redrawing a portion of the contour, for example, using a brush tool, which will be discussed in more detail later. In step 212, refinement module 120 recalculates the contour based on the user input received via UI module 116 through user input device 108. In step 214, rendering module 114 outputs the recalculated contour determined by refinement module 120 via display 110.

[0036] Figures 3A to 3E An initial segmentation method for segmenting an image, which can be performed, for example, in step 206, is illustrated. This method is performed by segmentation module 118 based on user input received from the user via user input device 114 using UI tools provided by UI module 116. It should be understood that... Figures 3A to 3E The specific images shown are merely illustrative and not restrictive.

[0037] Figure 3A An image captured without segmentation is shown, such as the image initially captured in step 202 and displayed in step 204. This image includes the (unsegmented) object 302 and a background region 303 outside the object 302. In this example, the image is a medical scan image through the pelvis, and the object 302 is the bladder.

[0038] Figure 3B An example of a UI drawing tool 305 that can be provided by UI module 116 is shown. In this case, tool 305 takes the form of a brush tool (referred to herein as an "uncertainty brush"). A brush tool is a drawing tool with a finite (non-zero) width or area greater than a single point or pixel, i.e., a non-point tool.

[0039] Figure 3C The use of the uncertainty brush 305 is illustrated. The user uses the uncertainty brush tool 305 (e.g., using a mouse on a clickable interface, or a finger or stylus on a touchscreen) to paint an uncertainty area 306 around the perimeter of the object 302. The brush tool 305 leaves a trail in the form of the uncertainty area 306 superimposed on each painted area of ​​the image.

[0040] Figure 3D This shows the result after the user has painted around the entire perimeter of object 302. The uncertainty area 306 thus forms a complete loop around object 302, which has a finite width (wider than the lines).

[0041] The uncertainty region 306 defines a region that can be either foreground or background. The segmentation module 118 automatically classifies the region outside the ring formed by the uncertainty region 306 as known background, the region inside the ring formed by the uncertainty region 306 as known foreground, and the region of the uncertainty region 306 itself as undetermined (currently unknown). For example, this can be achieved by determining a pixel mask above the image, where each pixel outside the ring defined by the uncertainty region 306 is mapped to a first value (e.g., 1) indicating background 303, each pixel inside the ring defined by the uncertainty region 306 is mapped to a second value (e.g., 0) indicating foreground 302, and each pixel in the uncertainty region 306 itself is mapped to a third value indicating uncertainty.

[0042] Figure 3E The resulting segmentation is shown, including the contour (i.e. shape) 304 automatically determined by the segmentation module 118 based on the classified foreground region 302, background region 303, and uncertainty region 306. This can be determined based on geodesic distance.

[0043] Specifically, the segmentation module automatically scans each of a plurality of points within the uncertainty region 305 (e.g., each pixel thus marked in the mask). For each such point (e.g., each pixel), it calculates the shortest geodesic distance to the background 303 and the shortest geodesic distance to the foreground 302. Which one is the shortest determines whether the uncertainty point (e.g., the pixel in question) is reclassified as foreground or background: if the shortest geodesic distance to the foreground 302 is the shortest, it is reclassified from uncertainty as foreground; but if the shortest geodesic distance to the background 303 is the shortest, it is reclassified from uncertainty as background.

[0044] The shortest geodesic distance from a point (e.g., a pixel) in uncertainty region 305 to background 303 can be determined by comparing the geodesic distances from that point to multiple candidate points (e.g., pixels) in background 303 (i.e., the region already classified as known background). For efficiency, it is not necessary to compare the distance to every possible point in background 303. Therefore, note that for the purposes of this application, the shortest geodesic distance to background 303 means the shortest distance among those compared.

[0045] Similarly, the shortest geodesic distance from a point (e.g., a pixel) in uncertainty region 305 to foreground 302 can be determined by comparing the geodesic distances from that point to multiple candidate points (e.g., pixels) in foreground 302 (i.e., the region already classified as a known foreground). For efficiency, it is not necessary to compare the distance to every possible point in foreground 302. Therefore, note that for the purposes of this application, the shortest geodesic distance to foreground 302 means the shortest path among the compared paths.

[0046] For example, segmentation may only compare the distances between foreground and background pixels in a limited candidate region surrounding uncertainty region 306, and / or may only compare the distances between every n pixels in the foreground or background.

[0047] Geodesic distance is a distance metric in which the distance is weighted by one or more parameters (such as intensity) of each point or pixel in an image along the path between the point in question and the foreground or background. The concept of geodesic distance itself is known in the art.

[0048] For example, given an image I defined on the 2D domain Ψ, a binary mask M (where... ) and foreground or background area Ω (where The unsigned geodesic distance between each compared pixel x and Ω is defined as:

[0049] Among them (1)

[0050]

[0051] Where x' is a pixel or point different from x; P a，b It is the set of all considered paths between points a and b; and This indicates a path where s∈[0,1]. The labels a and b are simply different parameter names used to define d(x, x'). Two different labels are used because the equation above has two independent functions. (Symbols) Let ||...|| represent the gradient, and ||...|| are the magnitudes. The spatial derivative Γ′(s) is... The unit vector u = Γ′(s) / ||Γ′(s)|| is tangent to the direction of the path. The factor γ weights the contribution of the image gradient to the spatial distance. When γ = 0, the distance D in equation (1) decreases to the traditional Euclidean path length.

[0052] I is a parameter representing the cost or weight of the path at each point, analogous to altitude on a map. That is, the properties or characteristics considered by geodesic distance distinguish it from simple linear (Euclidean) distance. In embodiments, I can be, for example, achromatic pixel intensity (e.g., luminance). However, it can be another attribute, such as chromaticity value; or the intensity of the red, green, or blue (RGB) channels. As another possibility, I can be some other technical parameter of MRI scans, CT scans, etc., indicating attributes beyond those visible in the image itself (e.g., attributes related to the composition of the scanned material). In some embodiments, I can be a composite measure considering more than one of the above parameters. The choice of parameters or measures depends on the design of the desired application.

[0053] In other words, geodesic distance can be defined as:

[0054]

[0055] Where v is a pixel or point in the image, and ρ is the spacing ratio in that direction. The reason for ρ is that if the image voxels (e.g., in some medical images) are not square, then the spacing ratio in a given direction is incorporated into the distance calculation to account for this. For example, this can be implemented as follows.

[0056] private void CalculateSpacingScale <t>(Volume3D <t>volume)

[0057]

[0058] To avoid any doubt, attribute I is naturally taken from "below" the image, or from the visible UI overlay of the trajectory without the uncertainty region 306, for example, in Figure 3C and Figure 3D This is shown to the user. It's simply a graphical UI element showing the user where the uncertainty region 306 is being painted. The graphical UI representation of the uncertainty region 306 is not part of the pixel values ​​processed to determine the segmentation. Only the underlying pixel mask or classification mapped to the uncertainty region is used in this calculation.

[0059] Once the segmentation module 118 has reclassified all processed points (e.g., each pixel) in the uncertainty region 305 as foreground or background, the contour (shape) 304 around the foreground object 302 can thus be determined. If each pixel is reclassified, the contour 403 can simply be the outermost pixel of the foreground object 302 or the innermost pixel of the background region 303, or potentially a fit between the foreground region 302 and the background region 303 (the contour 304 can be defined with subpixel precision, where whether each pixel is foreground or background depends on whether most of the pixel's area falls inside or outside the foreground). On the other hand, if only geodesic distance comparisons are performed for points representing a subset of pixels in the uncertainty region 305, then the contour 304 can be determined as the best fit between the processed points.

[0060] refer to Figures 3A to 3E The described method thus provides a semi-automated process for segmenting images. Note that this is merely one example of a method for segmenting images. In other embodiments, object recognition algorithms can be used to perform the initial segmentation for determining contour 304 in a fully automated manner, such as those disclosed in US10,235,605. In other embodiments, the initial segmentation can be performed in a fully manual manner, for example by a user using a manually drawn line tool (a dotted tool, rather than a brush tool with a finite width or area).

[0061] Regardless of the method used to perform the segmentation, the initial profile 304 may not turn out exactly as expected. The automated portion of an automated or semi-automated process may not function correctly, or the user may make manual errors in the manual portion of a fully manual or semi-automated process.

[0062] There are various possible UI tools available for refining the contours around the image (e.g., in steps 210-214). The refinement module 120 can implement refinement based on input received by the user input device 108 using UI tools provided by the UI module 116.

[0063] Figure 4 An example tool is shown. Here, outline 304 is in the form of a polygon, and a set of nodes 401 are rendered around the vertices of the polygon. The user can then use the mouse pointer tool 402 to click and drag the nodes 401 to drag in or out the corresponding vertices of the polygon.

[0064] Figure 5 Another possible tool is shown. Here, the user is provided with a pushing tool 501. The user uses a user input device such as a mouse or stylus to place the pushing tool inside the object or foreground (within the outline 403), and then uses tool 501 to push outward on the outline. The outline 304 extends outward with the user's pushing action.

[0065] Figure 6 Another possible tool is shown. Here, the user uses the painting brush tool 601 to repaint the entire foreground area. Then, the new foreground 302 simply becomes whatever area the user has painted, and the new outline 304 is simply the outline around the area the user has newly painted.

[0066] Figure 7 The operation of another possible refinement tool is illustrated. For the sake of illustration, consider a scenario in which the initial segmentation 206 performed by segmentation module 118 produces... Figure 7 The error profile shown is 304, not 304. Figure 3E The correct contour shown. The disclosed tool allows the user to correct this by replacing a portion of the initial contour with user-drawn lines, defined at least partially manually, drawn on the image, to obtain, for example, a correct contour as shown. Figure 3E The outline shown. The user draws between the starting point 701 and the ending point 702 to define a line that replaces a segment of an existing outline 304.

[0067] In one embodiment, the user accomplishes this by using a brush tool to paint over the new uncertainty region 306' between the starting point 701 and the ending point 702. Advantageously, in some embodiments, this can be the same brush tool 305 used to perform the initial segmentation 206. Therefore, this method provides an efficient and intuitive way to correct the segmentation. The process then proceeds exactly as previously described... Figures 3A-3E It works as described, but only processes points (e.g., each pixel) within the uncertain area 306' on the newly painted surface around a portion of the perimeter of object 302, instead of... Figure 3D Points within the entire perimeter shown. Once the user has painted over this area 306', points (e.g., each pixel) within the new uncertainty area 306' are reclassified as uncertain. Points not in this new uncertainty area 306' retain their initial classification as known foreground or background from the initial segmentation step 206. In an embodiment, this may include defining a new pixel mask for the image, where pixels in the new uncertainty area 306' are relabeled with a third value representing uncertainty, but other pixels are still labeled with their previous first or second values ​​(e.g., 0 and 1) representing known background and foreground. Thus, note that points or pixels previously (for the purposes of initial segmentation 206) classified as "known" foreground or background may now be reclassified as uncertain.

[0068] Then, the refinement module 120 can use the same or similar geodesic distance algorithm to reclassify the points or pixels considered in the new uncertainty region 306' as foreground or background. That is, it scans each point considered (e.g., each pixel) and, for each point, determines the shortest geodesic distance to the known foreground 302 (based on the current mask or classification) and the shortest geodesic distance to the known background 302 (also based on the current mask or classification). If the shortest geodesic distance 302 to the foreground is shorter than the shortest geodesic distance to the background 303, the point / pixel in question is classified as foreground; and if the reverse is true, the point / pixel in question is classified as background.

[0069] In practice, refinement module 120 may include some of the same code as segmentation module 118, and it should be understood that... Figure 1 The diagram shown is slightly schematic.

[0070] Considering it through diagrams Figure 7 For example, note that "user-drawn lines" does not necessarily mean that the lines are drawn entirely manually. Rather, in this embodiment, it is drawn in a partially manual manner, where the user uses a brush with a non-dot width or area to draw a wider area 306' that contains the area where new lines will be determined, and then within the area 306' drawn by the user, the new lines are automatically determined by the thinning module 120.

[0071] Nevertheless, in other embodiments, it is not excluded that a dotted drawing tool (as opposed to a brush) can be used to manually draw new lines that replace a portion of the initial outline 304.

[0072] In this embodiment, the user begins at the edge of the existing contour 304. The user then "paints" onto any point on the image. However, the endpoint 702 does not need to be a point on the existing contour 304. Instead, a line can be automatically calculated from the endpoint 702 of the user interaction to the closest point on the contour that the user has consistently interacted with. This can be done, for example, by calculating the distance from the endpoint of the user interaction to every point on the contour, such as a distance relative to the geodesic distance based on the Pythagorean theorem (x). 2 +y 2 =z 2 The straight-line distance. Figure 8 It shows the endpoint that is not on the outline but connects to the nearest point.

[0073] Because calculating the point closest to the endpoint on the contour is computationally expensive, the embodiments may optionally employ one or more efficient methods to improve performance. One such method is to not calculate the square root of the result, but instead calculate it by x. 2 +y 2 Sort the distances. Regardless of the square root, the closest point will yield the smallest result. A second, potentially efficient approach is to skip multiple points based on the contour's dimensions instead of calculating the distance to the endpoint for each point on the contour. This will result in an approximation of the closest point, but the larger the contour size, the smaller the impact of this approximation on the final result.

[0074] In a variation of the above, the distance from the user's endpoint to profile 304 can instead be determined based on the geodesic distance. In this case, the line connecting the user's endpoint to the initial profile 304 will be the shortest geodesic distance from the user's endpoint to the initial profile.

[0075] In one embodiment, the user is constrained to begin drawing a line at point 701 on the initial contour 304 determined by the segmentation module 118 in step 206. Alternatively, a similar technique as described above can be used to allow the user's starting point 701 and ending point 702 to begin at a distance from the initial contour 304. In other embodiments, the user's starting point can be allowed to begin at a distance from the initial contour, but the ending point can be constrained to end on the initial contour, or the user can be constrained to begin and end on the initial contour.

[0076] The refinement module 120 determines refinement lines that replace a portion (but not all) of the initial contour 304 determined by the segmentation module 118 in the initial segmentation step 206. As described, the refinement replacement lines may consist solely of user-drawn lines, or they may consist of user-drawn lines and one or more automatically generated connecting lines that connect the start and / or end points of the user's lines to the initial contour 304.

[0077] The refinement module 120 can also reclassify which regions of the image are foreground objects 302 and which regions are background images 303.

[0078] Notice, Figure 7 The scenario depicted actually presents two possible solutions: In principle, the user might intentionally keep the loop or lobe in the upper left and classify that smaller area as foreground; exclude the larger lines around the bladder and classify the larger bladder area as background. However, in practice, the user would naturally want to keep the larger loop around the bladder as outline 304 and classify the area within the loop (bladder 302) as foreground; exclude the smaller, incorrect loop in the upper right and classify that area, along with the rest of the image's boundary areas, as background 303. There are various ways to configure the thinning module 120 to handle this potential blurring.

[0079] In one embodiment, the refinement module 120 can be simply configured to determine two solutions and offer them to the user. As a variation, the refinement module 120 can provide the user with an option to choose which of the two solutions to accept via the UI module 116. In other embodiments, the refinement module 120 can be configured to automatically select between two possible solutions. In such an embodiment, if the automatically selected solution is not what the user wants, the refinement module 120 can provide the user with an option to switch to another solution via the UI module 116.

[0080] One possible way for the thinning module to perform automatic selection 120 is to calculate and compare the foreground area within the new contour 304 of the two solutions, and select the solution with the larger foreground area as the desired solution (or, equivalently, if only one object is segmented, calculate and compare the background area of ​​the two solutions, and select the solution with the smaller background area). As another example of making a selection, the thinning module 120 can select which solution might be desired based on the history of user interactions in current and / or past images. For example, in the case where the initial segmentation 206 is performed in at least partially manual manner, the thinning module 120 can select the area closest to the contour 304 or brush stroke initially drawn by the user in the same image as the foreground. Alternatively, it can make the selection based on the history of manual image segmentation and / or thinning performed in previous images (e.g., using machine learning algorithms such as neural networks). A combination of the above methods can also be used.

[0081] The following describes an example algorithm for automatically selecting between two possible solutions based on area. After the start and end points on the contour have been calculated, the algorithm then calculates what the user is trying to achieve with a brush stroke. As discussed, this type of user interaction can be ambiguous for a computer algorithm, and therefore in such an embodiment, the algorithm must calculate which contour shape (i.e., outline) is expected by calculating two sets of possible contours. The first contour begins at either the start or end point of the brush interaction. The algorithm then travels clockwise along the edges of the contour to the start or end point of the brush interaction. The other possible contour begins at the same point but travels counterclockwise along the edges of the contour until it reaches the same end point.

[0082] Once two contours are created, the algorithm calculates the size of each calculated region within each contour by counting the number of pixels or voxels within each closed contour. The resulting segment is selected as the largest of the two regions. This works for many different use cases, some of which are even quite vague when the user wants to perform a specific action (such as drawing a line in the middle of an existing contour, etc.). Figure 9 (As shown).

[0083] When the start and end points are the same (or within x pixels or the same as each other), a secondary edge case is also introduced in the algorithm. In this case, the user has already created a new closed contour (shape), and therefore the algorithm assumes this is the expected shape. In this case, the algorithm first calculates whether all interaction points are completely present in the existing segmentation. If true, the existing segmentation is completely replaced by the result of the new interaction. If some or all points are outside the existing contour, the algorithm assumes the user is adding to the existing contour. Therefore, the two contours are merged, and the new contour is extracted from the outer region of the merged result.

[0084] Regardless of any specific tools available for examining or refining 2D images, one can expect to find mechanisms for segmenting 3D images.

[0085] Figure 10 A method for segmenting a 3D image according to embodiments disclosed herein is illustrated.

[0086] The method begins at step 1002, capturing a 3D image via image capture device 102. The 3D image comprises an ordered set of 2D images. In an embodiment, the 3D image takes the form of a static 3D image, where each 2D image in the set ("stack") is a different parallel slice through the volume. In an alternative use case, the 3D image may take the form of 2D video (two-dimensional space), where each 2D image is a different temporal frame (the third dimension is time). As an illustration, the following will be given using examples of stacked 2D image slices forming a static 3D image, but it should be understood that this is not limiting. In an embodiment, step 1004 may include... Figure 2 Several instances of step 202 are discussed.

[0087] Each 2D slice of the 3D captured image is provided to the processing device 104 for processing. This may include storing the captured image slices in memory 106, and the processing may include processing the stored image slices from memory 106.

[0088] In step 1004, segmentation module 118 performs initial segmentation on a first image slice in the stacked 2D image slices to segment it into at least one foreground object and a background region. Rendering module 114 renders the first image slice along with its outline to the user via display device 110. In an embodiment, step 1004 may include performing initial segmentation on the first image slice from... Figure 2 The first instance and / or steps 204 to 208 Figures 3A-3E The division.

[0089] In step 1006, the user examines the outline determined in step 1004 and decides whether to approve it. If the outline is already acceptable to the user, the user can simply select to approve it via UI module 116 without modification. Alternatively, in this step, the user can choose to refine the outline in at least a partially manual manner. This can be done through previous discussions... Figures 4 to 9 The process can be carried out using any means discussed. Once refined to a level that satisfies the user, the user then selects and approves the refined outline via UI module 116 and user input device 108. In an embodiment, step 1006 may include performing a process from [unclear - likely a specific method or procedure] on the first 2D slice. Figure 2 The first instance of steps 210-214.

[0090] Once the user approves the outline in a slice, the slice (including its outline) can be described herein as "locked" or "locked in". In an embodiment, this means that all points (e.g., all pixels) in the slice that are available for processing are now classified as known foreground or known background (none are classified as uncertain).

[0091] In step 1008, segmentation module 118 performs initial segmentation on a second 2D image slice in the stacked 2D image slices to segment it into at least one foreground object and a background region. Rendering module 114 renders the first image slice along with its outline to the user via display device 110. In an embodiment, step 1004 may include performing a segmentation on the first image slice from... Figure 2 The second instance and / or steps 204 to 208 Figures 3A-3E The segmentation is as follows: The second image slice is not adjacent to the first image slice in the stack; that is, there are one or more intermediate slices between the first slice and the second slice, according to the order in which the slices appear in the stack.

[0092] In step 1010, the user examines the outline determined in step 1006 and decides whether to approve it. If the outline in the second slice is already acceptable to the user, the user can simply select to approve it via UI module 116 without modification. Alternatively, in this step, the user can choose to refine the outline of the second slice in at least a partially manual manner. This can be achieved through the preceding discussion of... Figures 4 to 9 The process can be carried out using any means discussed. Once refined to user satisfaction, the user then selects and approves the refined outline in the second slice via UI module 116 and user input device 108. In an embodiment, step 1006 may include performing a process from [unclear - likely a specific method or procedure] on the first 2D slice. Figure 2 The second instance of steps 210-214.

[0093] In an embodiment, the method may also include steps similar to 1004-1006 or 1008-1010, performing necessary corrections for a third image slice or more image slices (but not all slices in the stack).

[0094] In step 1012, the segmentation module performs 3D geodesic interpolation between the first 3D image slice and the second 3D image slice to segment the volume between them into foreground and background, wherein the 3D region where object 302 appears is classified as foreground. This will refer to Figure 11 A more detailed discussion follows. This could include determining the 3D contours around the object. In an embodiment, this could also include applying necessary corrections between the second and third slices, etc.

[0095] In step 1014, optionally, the determined 3D contour overlaid on the 3D image can be rendered to the user via the display device 110. See, for example, [link to relevant documentation]. Figure 12B .

[0096] In step 1016, the method may include applying the segmentation to a practical application. This may include, in an image or in the real world, aligning the process with or away from a segmented foreground object. For example, if the image is a medical scan and the object is a tumor, step 1016 may include aligning a radiotherapy treatment with the tumor inside a subject's body. Or, if the object is an organ, step 1016 may include aligning a radiotherapy treatment with or away from the organ.

[0097] Figure 11 The 3D geodesic interpolation performed in step 1012 is shown. Figure 11 A first 2D image slice Sa and a second 2D image slice Sb are shown, wherein in the stack, there are one or more intermediate slices Sab between them. It is possible that after the refinement described above, the first slice Sa and the second slice Sb are each locked, i.e., approved by the user. The intermediate slices Sab form intermediate bodies or regions of one or more unlocked slices, and wherein at least a portion of each intermediate slice is classified as indeterminate at the start of the 3D geodesic interpolation process. Figure 11 The contours Ca and Cb, which have been defined at this stage around object 302 in the first 2D slice and the second 2D slice, are also shown.

[0098] As an optional optimization, prior to 3G geodesic interpolation, some regions of one or more intermediate slices (Sab) can be pre-classified as known foreground, and / or some regions of one or more intermediate slices (Sab) can be pre-classified as known background. For example, any region within the contours Ca, Cb of the intermediate slices located in the image plane surrounding object 302 in the first and second slices can be pre-classified as known foreground. This can be thought of as a column defined by the xy pixel coordinates (coordinates in the image plane) of the images classified as foreground in the first and second slices. And / or similarly, any region outside the contours Ca, Cb of the intermediate slices located in the image plane in the first and second slices can be pre-classified as known foreground. All other regions are classified as uncertain. This optimization advantageously reduces the search space of the 3D geodesic interpolation algorithm, thereby reducing computational complexity. Alternatively, it could be simply to start with the entire region of each intermediate slice (Sab) classified as uncertain, but this would require more computational resources for the search.

[0099] In either case, to perform 3D geodesic interpolation, the algorithm operates in a similar manner to the previous discussion regarding individual 2D slices, but now in three dimensions instead of two. That is, it processes each of the multiple points Pi initially classified as uncertain at the start of the search. For each such point Pi, it determines the shortest geodesic distance from that point to the currently known foreground, and the shortest geodesic distance from that point to the currently known background; and classifies point Pi as foreground if the shortest foreground distance is shorter than the shortest background distance, but as background if the shortest background distance is shorter than the shortest foreground distance. However, the shortest distance can now be the distance from Pi to a point in one of the other image slices between the first and second image slices (instead of within the same image slice as in the purely 2D case). Therefore, this is referred to herein as a 3D geodesic distance, i.e., a path not constrained to be within the same 2D image slice.

[0100] Geodesic distance is also a distance that takes into account the "cost" or "weight" of one or more image attributes (such as pixel intensity) along the path. The formula is the same as in the 2D case, but now in 3D coordinates instead of 2D coordinates. Similarly, geodesic distance is a path-weighted distance between two points by one or more parameters (such as pixel intensity). Analogies to altitude on a map become more difficult to consider in the 3D case because the spatial coordinates (relative to the parameters) are now 3D. Instead, one can think of geodesic distance as the path of least resistance through a volume, where different regions of the volume have higher or lower resistance to travel than other regions.

[0101] In one embodiment, point Pi can be each voxel classified as uncertain in the intermediate region (i.e., each uncertain pixel in each intermediate slice within an intermediate slice). Alternatively, point Pi can be simply a smaller selection of points within the intermediate region.

[0102] Regardless of the approach, once all the points Pi to be processed have been classified as foreground or background, a 3D profile can be generated. If each voxel has been processed and is therefore now classified as a known background or foreground, the profile can simply be the outermost voxel of the foreground object or the innermost voxel of the background region, or potentially a fit between the foreground and background regions (the profile can be defined with sub-voxel precision). On the other hand, if only geodesic distance comparisons are performed for selected points representing a subset of voxels, the profile can be determined as the best fit between the processed points.

[0103] Figure 12A and Figure 12B An example is shown where the process is used to determine the 3D volume around an organ (in this case, a human bladder) in a medical scan. Figure 12A The first slice is shown, and Figure 12B The second slice and the resulting 2D contour 1200 are shown.

[0104] Figure 13 Another optional optimization that can be implemented according to the embodiments disclosed herein is illustrated. Here, at least three slices have contours initially defined within a 2D slice: a first slice Sa, a second slice Sb, and a third slice Sc (where Sb is between Sa and Sc). Similar to one or more intermediate slices Sab between the first slice Sa and the second slice Sb, there are also one or more additional intermediate slices Sbc between the second slice Sb and the third slice Sc. Each of the first, second, and third slices is locked, that is, each region within each of these slices is classified as foreground or background. In an embodiment, 3D geodesic interpolation calculation may include a first geodesic interpolation performed between the first slice Sa and the second slice Sb; and a second 3D geodesic interpolation, independent of the first 3D geodesic interpolation, between the second slice Sb and the third slice Sc. This means that in the first 3D geodesic distance calculation, when calculating the shortest geodesic distance between Pi and the foreground and background, the possible foreground and background points where the path from Pi terminates are constrained to points between the first and second image slices (including both slices) (therefore, as far as the first or second slice, but not exceeding it). Similarly, in the second 3D geodesic distance calculation, when calculating the shortest geodesic distance between Pi and the foreground and background, the possible foreground and background points where the path from Pi terminates are constrained to points between the second and third image slices (including both slices).

[0105] In some such embodiments, the segmentation module 118 can be configured to automatically select slice pairs from which independent 3D geodesic interpolation operations are performed. For this purpose, preferably, it will automatically select the locked slice pairs that are closest to each other in stacking order. In other words, each locked slice pair consists of a locked slice and the next closest locked slice along the stack. Therefore, refer to... Figure 13 Starting with Sa, the next closest locked slice is Sb, so Sa and Sb are automatically selected as a pair. Then, starting with Sb, the next closest slice in the same direction in the stack is Sc, so Sb and Sc are selected as another pair, and so on. In other words, among those locked slices, locked slice pairs are locked slices that are adjacent (nearby) in the stack order (but still not adjacent to each other in the stack, i.e., between the two locked slices in each given locked slice pair, there are one or more unlocked slices).

[0106] If the user subsequently modifies the contour in one of the slices, only the one or two individual 3D regions affected by that modification will be recalculated. Therefore, suppose the user subsequently modifies the third slice Sc (which was previously "locked"). The algorithm will recalculate the 3D segmentation between the two slices by performing a 3D geodesic distance calculation only between the second slice Sb and the third slice Sc; however, it will not recalculate the 3D segmentation between the first and second slices. If there is a fourth slice after the third slice, it will recalculate between the third and fourth slices. However, if there is a fifth slice after the fourth slice, it will not recalculate between the fourth and fifth slices.

[0107] exist Figure 14 Another optional feature, illustrated by way of example, provides the user with the option to switch between linear 3D interpolation and geodesic 3D interpolation. UI module 116 presents, for example, a number of UI options (UI controls) selectable by the user via display 110. These may include a UI control 1402 for selecting linear 3D interpolation and another UI control 1403 for selecting geodesic 3D interpolation ("smart" interpolation). In some cases, the controls may also include a control 1401 for disabling 3D interpolation. Such UI controls 1401, 1402, and / or 1403 may take the form, for example, clickable or touchable on-screen buttons, or options in drop-down menus, etc.

[0108] If the user selects (activates) the geodesic 3D interpolation control 1403, the UI module 116 controls the segmentation module 118 to switch to geodesic 3D interpolation mode, in which it (e.g., according to the above embodiment) performs interpolation between slices using 3D geodesic interpolation. On the other hand, if the user selects (activates) the linear 3D interpolation control 1402, the UI module 116 controls the segmentation module 118 to switch to linear 3D interpolation mode, in which it performs interpolation between slices in a linear manner. This can be between one or more pairs of slices, similar to the geodesic mode. In some such embodiments, the 3D linear interpolation control 1402 and the 3D geodesic interpolation control 1403 may allow the user to switch between performing interpolation in two different modes for the same locked set of slices (the same one or more pairs) (e.g., Sa&Sb or Sa&Sb and Sb&Sc). The rendering module 114 can be configured to switch between rendering 3D contours on a 3D image via display 110 in two different modes for the same set of slices in response to the user's selection of two different corresponding controls 1402, 1403. In this way, the user can compare the results of two different types of interpolation and (e.g., in the case of a physician examining a body scan, based on his / her professional experience) select the one that best matches his / her expectations.

[0109] Note that even in linear 3D interpolation mode, in the embodiments, 2D geodesic techniques can still be used to determine the 2D contour within each individual locked slice, for example, as per [reference to...]. Figures 3A-3D and / or Figure 7 The subject of discussion.

[0110] It should be understood that the above embodiments are described by way of example only.

[0111] More generally, according to one aspect disclosed herein, a computer-implemented method is provided, comprising: receiving a 3D image including objects depicted in the image, the 3D image comprising an ordered set of 2D images; determining contours surrounding objects in a first 2D image of the 2D images; and determining contours surrounding objects in a second 2D image of the 2D images, the second 2D image being non-adjacent to the first 2D image in the ordered set, the ordered set having an intermediate region between the first and second 2D images comprising one or more intermediate 2D images of the 2D images. In each of the first and second 2D images, the interior of the contour is classified as foreground, and the exterior of the contour is classified as background. The method also includes performing 3D geodesic distance calculations, including, for each of a plurality of points in the intermediate region: calculating the shortest foreground distance and calculating the shortest background distance, wherein the shortest foreground distance is the shortest 3D geodesic distance from the point to the foreground and the shortest background distance is the shortest 3D geodesic distance from the point to the background; and classifying the point as foreground if the shortest foreground distance is shorter than the shortest background distance, but classifying the point as background if the shortest background distance is shorter than the shortest foreground distance.

[0112] In an embodiment, the method may include: after classifying the plurality of points in the intermediate region, determining a 3D contour around an object in the intermediate region by determining the contour between points classified as foreground and points classified as background.

[0113] In an embodiment, determining the contours in one or both of the first 2D image and the second 2D image may include at least partially automatically determining the contours.

[0114] In some such embodiments, determining the contours in one or both of the first 2D image and the second 2D image may include determining the contours in a partially automatic manner by:

[0115] - Receive user input that defines an uncertain region around the perimeter of an object in a 2D image, wherein the interior of the uncertain region is classified as foreground and the exterior of the uncertain region is classified as background; and

[0116] - For each of multiple points in a 2D image, determine the shortest 2D geodesic distance from that point to the foreground and the shortest 2D geodesic distance from that point to the background within the 2D image. If the shortest 2D geodesic distance to the foreground is shorter than the shortest 2D geodesic distance to the background, then classify that point as foreground. However, if the shortest 2D geodesic distance to the background is shorter than the shortest 2D geodesic distance to the background, then classify that point as background.

[0117] In an embodiment, determining the outline in one or both of the first 2D images may include: the user manually refining the outline via a UI tool.

[0118] In an embodiment, determining the contour in one or both of the first 2D image and the second 2D image may include: displaying the contour to a user and receiving the user's approval for locking the contour.

[0119] In some such embodiments, the method may include: receiving user approval to lock contours around more than two 2D images in the set; and automatically selecting locked 2D image pairs, performing instances of 3D geodesic interpolation between the locked 2D image pairs. Each automatically selected locked 2D image pair may consist of one locked 2D image and the next nearest locked 2D image within an ordered set, wherein the first 2D image and the second 2D image are one of the locked 2D image pairs.

[0120] In one embodiment, prior to the 3D geodesic distance calculation between the first and second 2D images, the method may include: performing an initial classification based on the fact that a region in each intermediate 2D image lies within the contours of both the first and second 2D images in the common plane of the set of 2D images, classifying that region as foreground; and / or, based on the fact that a region in each intermediate 2D image lies outside the contours of both the first and second 2D images in the plane, classifying that region as background. In this case, the plurality of points in the 3D geodesic distance calculation are points that were neither classified as foreground nor background in the initial classification.

[0121] In an embodiment, the method may include: determining the contours surrounding an object in a third 2D image within the 2D image, the third 2D image being outside the intermediate region and not adjacent to the first and second regions in the ordered set, the ordered set having an additional intermediate region between the second and third 2D images including one or more additional intermediate 2D images from the 2D images; and performing a further instance of calculating the 3D geodesic distance between the second and third 2D images independently of the calculation of the 3D geodesic distance between the first and second 2D images.

[0122] In some such embodiments, the method may include: re-performing the 3D geodesic distance calculation between the second 2D image and the third 2D image in response to a user subsequently modifying the contour in the third 2D image, without re-performing the 3D geodesic distance calculation between the first 2D image and the second 2D image.

[0123] In an embodiment, the geodesic distance can be a distance metric weighted by at least pixel intensity along a path between a point and the foreground or background. Alternatively or additionally, it can be a distance metric weighted by one or more other pixel attributes or image attributes along that path.

[0124] In an embodiment, the intermediate region may include an intermediate body comprising a plurality of the 2D images within the ordered set between the first 2D image and the second 2D image.

[0125] In an embodiment, the method may include providing a user interface option that allows a user to switch between 3D geodesic interpolation between the first and second slices and linear 3D interpolation between the first and second slices.

[0126] In an embodiment, the 3D image may be a static 3D image volume, wherein each of the 2D images in the set is a different 2D slice passing through the image volume.

[0127] For example, a 3D image may include a scan of at least a part of an organism’s body, and the object may be an organ or tumor within the body.

[0128] Alternatively, the 3D image may be a video image with two spatial dimensions, wherein each of the 2D images in the set is a different temporal frame.

[0129] According to another aspect disclosed herein, a computer program is provided, embodied on a computer-readable storage device, and configured to perform the methods described in any embodiment disclosed herein when run on one or more processors.

[0130] According to another aspect, a system is provided, comprising: a memory including one or more memory devices; and a processing means including one or more processing units; wherein the memory stores code arranged to run on the processing means, the code being configured to cause, when executed, to perform the methods described according to any embodiment disclosed herein.

[0131] Once the disclosure herein is given, other variations or use cases of the disclosed technology may become apparent to those skilled in the art. The scope of this disclosure is not limited to the described embodiments, but only to the appended claims.< / t> < / t>

Claims

1. A computer-implemented method, comprising: Receive a 3D image, the 3D image including objects depicted in the image, the 3D image comprising an ordered set of 2D images; Determine the contours surrounding the object in the first 2D image of the 2D image; and Determine the outline around the object in the second 2D image in the 2D image, the second 2D image being non-adjacent to the first 2D image in the ordered set, and having an intermediate region including one or more intermediate 2D images in the ordered set between the first 2D image and the second 2D image; In each of the first and second 2D images, the interior of the contour is classified as foreground, and the exterior of the contour is classified as background, and the method further includes: Performing 3D geodesic distance calculations includes, for each of a plurality of points in the intermediate region: calculating the shortest foreground distance and calculating the shortest background distance, wherein the shortest foreground distance is the shortest 3D geodesic distance from the point to the foreground, and the shortest background distance is the shortest 3D geodesic distance from the point to the background; and classifying the point as foreground if the shortest foreground distance is shorter than the shortest background distance, but classifying the point as background if the shortest background distance is shorter than the shortest foreground distance. Prior to calculating the 3D geodesic distance between the first 2D image and the second 2D image, the method further includes: An initial classification is performed, classifying a region as foreground based on the fact that a region in each of the intermediate 2D images lies within the common plane of the set of 2D images and within the contours of both the first and second 2D images; and / or, classifying a region as background based on the fact that a region in each of the intermediate 2D images lies outside the contours of both the first and second 2D images within the plane, wherein the plurality of points in the 3D geodesic distance calculation are points that were neither classified as foreground nor background in the initial classification.

2. The method according to claim 1, comprising: After classifying the plurality of points in the intermediate region, the 3D contour around the object in the intermediate region is determined by determining the contour between the points classified as foreground and the points classified as background.

3. The method of claim 1, wherein determining the contour in one or both of the first 2D image and the second 2D image comprises: The contour is determined automatically, at least partially.

4. The method of claim 3, wherein determining the contour in one or both of the first 2D image and the second 2D image comprises determining the contour in a partially automatic manner by: - Receive user input, the user input defining an uncertain region around the perimeter of the object in the 2D image, wherein the interior of the uncertain region is classified as foreground and the exterior of the uncertain region is classified as background; and For each of a plurality of points in the 2D image, determine the shortest 2D geodesic distance from the point to the foreground within the 2D image, determine the shortest 2D geodesic distance from the point to the background within the 2D image, and classify the point as foreground if the shortest 2D geodesic distance to the foreground is shorter than the shortest 2D geodesic distance to the background, but classify the point as background if the shortest 2D geodesic distance to the background is shorter than the shortest 2D geodesic distance to the background.

5. The method according to any one of claims 1 to 4, wherein determining the contour in one or both of the first 2D images comprises: Users can manually refine the outline using UI tools.

6. The method according to any one of claims 1 to 4, wherein determining the contour in one or both of the first 2D image and the second 2D image comprises: The outline is displayed to the user, and the user's approval to lock the outline is received.

7. The method of claim 6, comprising: The system receives user approval to lock the contours around more than two 2D images in the set, and automatically selects locked 2D image pairs, performing the 3D geodesic interpolation between the locked 2D image pairs, wherein each automatically selected locked 2D image pair consists of one locked 2D image in the ordered set and the next closest locked 2D image in the locked 2D image pair, the first 2D image and the second 2D image being one of the locked 2D image pairs.

8. The method according to any one of claims 1 to 4, comprising: Determine the outline around the object in a third 2D image within the 2D image, the third 2D image being outside the intermediate region and not adjacent to the first and second regions in the ordered set, having an additional intermediate region within the ordered set between the second and third 2D images, including one or more additional intermediate 2D images in the 2D images; as well as An additional instance of performing the 3D geodesic distance calculation between the second 2D image and the third 2D image, independent of the calculation of the 3D geodesic distance between the first 2D image and the second 2D image.

9. The method of claim 8, comprising: In response to a user subsequently modifying the contour in the third 2D image, the 3D geodesic distance calculation between the second 2D image and the third 2D image is re-executed, without re-executing the 3D geodesic distance calculation between the first 2D image and the second 2D image.

10. The method according to any one of claims 1 to 4, wherein the geodesic distance is a distance metric weighted by at least pixel intensity along the path between the point and the foreground or the background.

11. The method according to any one of claims 1 to 4, wherein the intermediate region includes an intermediate body, the intermediate body comprising a plurality of the 2D images within the ordered set between the first 2D image and the second 2D image.

12. The method according to any one of claims 1 to 4, comprising: A user interface option is provided that allows the user to switch between the 3D geodesic interpolation between the first 2D image and the second 2D image and the linear 3D interpolation between the first 2D image and the second 2D image.

13. A computer program product embodied on a computer-readable storage device and configured to perform actions when executed on one or more processors, the actions comprising: Receive a 3D image, the 3D image including objects depicted in the image, the 3D image comprising an ordered set of 2D images; Determine the contours surrounding the object in the first 2D image of the 2D image; and Determine the outline around the object in the second 2D image in the 2D image, the second 2D image being non-adjacent to the first 2D image in the ordered set, and having an intermediate region including one or more intermediate 2D images in the ordered set between the first 2D image and the second 2D image; In each of the first and second 2D images, the interior of the contour is classified as foreground, and the exterior of the contour is classified as background, and the action further includes: Performing 3D geodesic distance calculations includes, for each of a plurality of points in the intermediate region: calculating the shortest foreground distance and calculating the shortest background distance, wherein the shortest foreground distance is the shortest 3D geodesic distance from the point to the foreground, and the shortest background distance is the shortest 3D geodesic distance from the point to the background; and classifying the point as foreground if the shortest foreground distance is shorter than the shortest background distance, but classifying the point as background if the shortest background distance is shorter than the shortest foreground distance. Before calculating the 3D geodesic distance between the first 2D image and the second 2D image, the action further includes: An initial classification is performed, classifying a region as foreground based on the fact that a region in each of the intermediate 2D images lies within the common plane of the set of 2D images and within the contours of both the first and second 2D images; and / or, classifying a region as background based on the fact that a region in each of the intermediate 2D images lies outside the contours of both the first and second 2D images within the plane, wherein the plurality of points in the 3D geodesic distance calculation are points that were neither classified as foreground nor background in the initial classification.

14. The computer program product of claim 13, wherein the computer program product is configured to, when run on one or more processors, also perform actions, the actions comprising: After classifying the plurality of points in the intermediate region, the 3D contour around the object in the intermediate region is determined by determining the contour between the points classified as foreground and the points classified as background.

15. The computer program product of claim 13, wherein determining the contour in one or both of the first 2D image and the second 2D image comprises: The contour is determined automatically, at least partially.

16. The computer program product of claim 15, wherein determining the contour in one or both of the first 2D image and the second 2D image comprises determining the contour in a partially automatic manner by: - Receive user input, the user input defining an uncertain region around the perimeter of the object in the 2D image, wherein the interior of the uncertain region is classified as foreground and the exterior of the uncertain region is classified as background; and For each of a plurality of points in the 2D image, determine the shortest 2D geodesic distance from the point to the foreground within the 2D image, determine the shortest 2D geodesic distance from the point to the background within the 2D image, and classify the point as foreground if the shortest 2D geodesic distance to the foreground is shorter than the shortest 2D geodesic distance to the background, but classify the point as background if the shortest 2D geodesic distance to the background is shorter than the shortest 2D geodesic distance to the background.

17. The computer program product according to any one of claims 13 to 16, wherein determining the contour in one or both of the first 2D images comprises: Users can manually refine the outline using UI tools.

18. The computer program product according to any one of claims 13 to 16, wherein determining the contour in one or both of the first 2D image and the second 2D image comprises: The outline is displayed to the user, and the user's approval to lock the outline is received.

19. A system for image segmentation, comprising: - A memory, the memory comprising one or more memory devices; as well as - A processing apparatus, the processing apparatus comprising one or more processing units; The memory stores code arranged to run on the processing device, the code being configured to perform actions when executed, the actions including: Receive a 3D image, the 3D image including objects depicted in the image, the 3D image comprising an ordered set of 2D images; Determine the contours surrounding the object in the first 2D image of the 2D image; and Determine the outline around the object in the second 2D image in the 2D image, the second 2D image being non-adjacent to the first 2D image in the ordered set, and having an intermediate region including one or more intermediate 2D images in the ordered set between the first 2D image and the second 2D image; In each of the first and second 2D images, the interior of the contour is classified as foreground, and the exterior of the contour is classified as background, and the action further includes: Performing 3D geodesic distance calculations includes, for each of a plurality of points in the intermediate region: calculating the shortest foreground distance and calculating the shortest background distance, wherein the shortest foreground distance is the shortest 3D geodesic distance from the point to the foreground, and the shortest background distance is the shortest 3D geodesic distance from the point to the background; and classifying the point as foreground if the shortest foreground distance is shorter than the shortest background distance, but classifying the point as background if the shortest background distance is shorter than the shortest foreground distance. Before calculating the 3D geodesic distance between the first 2D image and the second 2D image, the action further includes: An initial classification is performed, classifying a region as foreground based on the fact that a region in each of the intermediate 2D images lies within the common plane of the set of 2D images and within the contours of both the first and second 2D images; and / or, classifying a region as background based on the fact that a region in each of the intermediate 2D images lies outside the contours of both the first and second 2D images within the plane, wherein the plurality of points in the 3D geodesic distance calculation are points that were neither classified as foreground nor background in the initial classification.

Images