System and method for segmenting objects in image data

Abstract

Systems and computer-implemented methods for segmenting an object in image data are provided. For this purpose, a segmentation result of the object is obtained, the segmentation result comprising an image element in the image data which may be a portion of the object, the image element which may be a portion of the object being determined by a probability value of the respective image element exceeding a probability threshold. More specifically, during operation of the system and the method, a first segmentation result of the object is obtained using a first probability threshold, and a second segmentation result of the object is obtained using a second probability threshold. The two segmentation results are visualized, which provides a user with a visual indication of the degree and way to which the segmentation results change according to confidence. In this manner, qualitative feedback is provided to the user regarding the segmentation results, which may improve the user's trust of the segmentation results and facilitate the user to employ the segmentation results.

Description

Technical Field

[0001] The present invention relates to a system and a computer-implemented method for segmenting objects in image data. The present invention also relates to a computer-readable medium comprising instructions for a computer program, the computer program comprising instructions for causing a processor system to execute the computer-implemented method. Background Art

[0002] Object segmentation is used in many application areas. For example, in the automotive field, object segmentation is used to detect obstacles on the road ahead, and in the security field, object segmentation is used to detect intruders, etc. Object segmentation is increasingly based on artificial intelligence (AI). For example, a machine learning model can be trained based on training data to learn to classify individual pixels or voxels as belonging to a specific object (e.g., belonging to a cat or a dog or the background).

[0003] Although segmentation models based on machine learning (or generally AI-based segmentation models) show great potential, the adoption rate of such models in many application areas is still very low. One example is the clinical field, where the adoption rate of such models in clinical practice is still very low. One of the reasons for the lack of adoption is the trust level of clinicians in AI-based segmentation models. On the one hand, clinicians may over-trust AI-based segmentation models, which may lead to errors that would not occur without AI. On the other hand, clinicians may distrust AI-based segmentation models, which may lead to suboptimal use of AI. However, in an ideal situation, clinicians would have an appropriate level of trust, which means that the trust level matches the actual performance of the AI-based segmentation model at its current stage of development.

[0004] It has been proposed that it is helpful to instill an appropriate level of trust in users (e.g., clinicians or radiologists) by communicating the uncertainty of an AI-based segmentation model to the users. Proper implementation of the communication of such AI model uncertainty may indeed have several advantages. First, the communicated uncertainty can imply to what extent the model can be trusted. Second, in the clinical field, communicating uncertainty can support the conversation between radiologists, surgeons, and patients and / or between medical experts, thus enabling better assessment of potential risks. Third, recent research has shown that by communicating uncertainty, the negative impact of AI model errors may be reduced due to increased situational awareness and increased understanding of the model [1]. Fourth, users can potentially reduce the uncertainty based on their domain knowledge through interface interaction. For example, by having a radiologist review and correct the AI output, the uncertainty of the AI-based tumor contour on a CT image may be reduced.

[0005] Reference [2] describes a method for determining the confidence level of image segmentation during a medical imaging process. The method is described as determining the confidence level associated with an image segmentation result in an application where there is no ground truth to compare with the segmentation result. The method includes visualizing the confidence levels of corresponding boundary portions of the segmented object.

[0006] Reference

[0007] [1] Siegling, L.B., “Uncertainty Communication by AI Assistants: The Effects on User Trust” (2020); [2] U.S. Patent US2021004965A1. Summary of the Invention

[0008] In a first aspect of the present invention, there is provided a system for segmenting an object in image data, comprising:

[0009] An input interface for accessing the image data;

[0010] A processor subsystem configured to:

[0011] Obtain a segmentation result of the object, wherein the segmentation result includes image elements in the image data that may be parts of the object, and the image elements that may be parts of the object are determined by the probability values of the corresponding image elements exceeding a probability threshold, and obtaining the segmentation result includes: obtaining a first segmentation result of the object using a first probability threshold, and obtaining a second segmentation result of the object using a second probability threshold; and

[0012] Generate display data, the display data including a visual representation of the first segmentation result and a visual representation of the second segmentation result; and

[0013] A display output interface for displaying the display data on a display.

[0014] In a further aspect of the present invention, there is provided a computer-implemented method for segmenting an object in image data, comprising:

[0015] Access the image data;

[0016] Obtain a segmentation result of the object, where the segmentation result includes image elements in the image data that may be part of the object, and the image elements that may be part of the object are determined by the probability value of the corresponding image element exceeding a probability threshold. Obtaining the segmentation result includes: using a first probability threshold to obtain a first segmentation result of the object, and using a second probability threshold to obtain a second segmentation result of the object;

[0017] Generate display data, where the display data includes a visual representation of the first segmentation result and a visual representation of the second segmentation result; and

[0018] Output the display data for display on a display.

[0019] In a further aspect of the present invention, there is provided a transient or non-transient computer-readable medium, the computer-readable medium including data representing a computer program, the computer program including instructions for causing a processor system to perform a computer-implemented method as described in this specification.

[0020] The above aspect of the present invention relates to accessing image data including an object to be segmented. The image data can be, for example, 3D image data (e.g., in the form of an image volume composed of voxels or a stack of 2D images), or 2D image data (e.g., a single 2D image). The object shown in the image data can be segmented (e.g., by applying classification techniques to the image data). Such classification techniques known per se can classify individual image elements of the image data as belonging to the object or not belonging to the object. More specifically, the classification by the classification technique can generate a probability map, which can indicate for the corresponding image element the probability that the image element belongs to the object. Thus, the classification technique can also be referred to as a probability classification technique. The probability can represent the certainty of the classification, because for example a probability of 0.0 (or 0%) or 1.0 (or 100%) can indicate that the classification technique is certain about its classification, while a probability between 0.0 and 1.0 can indicate the existence of uncertainty in the classification. Generally, if the probability is 0.5 or 50%, the uncertainty of the classification technique may be the highest. As a result of applying the classification technique to the image data, a probability map can be obtained. The probability map can indicate the position and shape of the object in the image data, because the map indicates which image elements may belong to the object.

[0021] The classification techniques described in the previous paragraphs are themselves known and can be based on machine learning, since applying the classification techniques can include applying a classification model of machine learning to the image data. Throughout the specification, references to classification techniques can thus be understood to include the use of classification models of machine learning. After obtaining the probability map as an output, a probability threshold can be used to determine which image elements have a high enough probability of belonging to the object. In this way, the probabilities of the probability map can be interpreted in a binary-like manner as "object" and "non-object" (or "background" or "other object"). Alternatively, the classification can provide the probabilities of several classes of objects (e.g., 3 classes: "tumor", "vessel", and "other"), in which case at least one probability threshold can be applied to the probabilities of at least one class of objects. The segmentation result of the object can be obtained by identifying a region that contains all or at least a large number of image elements classified as "object" (e.g., in the form of a line contour in 2D or as a 3D mesh that delineates the object boundary in 3D). In some embodiments, the object mask itself may already represent the segmentation result of the object, since the boundary of the object can be easily perceived in the object mask.

[0022] Note that although the use of classification techniques to obtain the segmentation result of the object is mentioned above, it is not necessary to use classification techniques. For example, instead of using classification techniques, probability segmentation techniques can be used to obtain the segmentation result. This probability segmentation technique can be directly controlled to output a segmentation result, where the image elements included in the segmentation result are considered "likely" to belong to the object, and the probability can be selected using a probability threshold parameter. By using two different parameter values, a first segmentation result and a second segmentation result can be obtained. Another example is that an initial segmentation result and a separate uncertainty map can be obtained. Based on this initial segmentation result and the uncertainty map, a further segmentation result can be derived. That is, the uncertainty map can indicate the probability that an image element belongs to a specific object. For this reason, a probability threshold can be used to derive one or more segmentation results based on the initial segmentation result. For example, if a lower probability is acceptable, the size of the initial segmentation result can be locally increased, or if a higher probability is required, the size of the initial segmentation result can be decreased.

[0023] According to the above measures, at least two segmentation results of the object can be obtained using different probability thresholds. More specifically, a first segmentation result can be obtained by only considering those image elements that belong to the object with a probability exceeding a first threshold, and a second segmentation result can be obtained by only considering those image elements that belong to the object with a probability exceeding a second threshold, and the two thresholds are different. Then, the two segmentation results can be visualized by generating a visual representation of the segmentation results and displaying these visual representations, for example, on a display.

[0024] Since the segmentation results are obtained using different probability thresholds, they can be considered to represent segmentation results with different levels of confidence. That is, when a lower probability threshold is used, image elements can be considered to belong to the object, but according to the probability map, the probability that these image elements actually belong to the object is low. In fact, the segmentation results obtained using a higher probability threshold represent smaller segmentation results that only include those image elements that actually have a relatively high probability of belonging to the object, while the segmentation results obtained using a lower threshold are typically wider segmentation results that include those image elements for which it is not very certain that they belong to the object. Typically, the segmentation results obtained using a higher probability threshold are included in the segmentation results obtained using a lower probability threshold (i.e., represent a subset thereof).

[0025] By visualizing the segmentation results to the user, the user is able to obtain a visual indication of the degree and manner in which the segmentation results change according to confidence. This is relevant for the following reasons. Near the boundary of the object, there is typically a transition region in which the probability of belonging to the object transitions from high (e.g., close to 1.0 or 100%) to low (e.g., close to 0.0 or 0%). The transition region itself can include intermediate probabilities (e.g., close to 0.5 or 50%), which can indicate low confidence. If the transition region is relatively wide, this can indicate that the exact boundary of the object is less certain. Conversely, if there is only a narrow transition region, this can indicate that the exact boundary of the object is relatively certain. By showing both visualizations, the shift of the object boundary can provide an indication of the width of the transition region, which can allow the user to perceive where the segmentation results are more or less certain. In particular, the user can obtain feedback on where along the boundary there is a high confidence in the boundary (e.g., a small shift in the boundary between the two segmentation results) and where there is a low confidence in the boundary (e.g., a large shift in the boundary between the two segmentation results). This can convey to the user where the local boundary is considered more certain and where the local boundary may be less certain. In this way, qualitative feedback on the segmentation results is provided to the user, which the user can use to, for example, accept or reject the segmentation results, correct the segmentation results, or understand whether the algorithm's (un)certainty matches the user's (un)certainty. Generally speaking, this quality feedback can increase the user's trust in the segmentation results and can facilitate the user's adoption of the segmentation results.

[0026] Although it is known to visualize the uncertainty in a segmentation result as a heatmap that can be overlaid on the image data, the heatmap typically obscures regions of interest in the image data (e.g., object boundaries). This may, for example, prevent a radiologist from validating the segmentation result in a region based on the underlying image data, which may be considered the ground truth data by a clinician. Additionally, while the uncertainty information itself can be visualized, the results thereof cannot. For example, the uncertainty information can be presented as a probability or transformed into probabilities of other modalities (e.g., color, transparency, size). This may require the user to interpret the results of the uncertainty themselves. The mental effort involved may increase the user's workload, and the interpreted results may be subjective. Additionally, most people lack a natural affinity for statistics and probability, thus hindering the accurate interpretation of the raw uncertainty information. By directly visualizing the segmentation results associated with different uncertainties, such an interpretation may not be required, as the results of the different uncertainties are directly visualized (i.e., this is achieved by the corresponding segmentation results being visualized).

[0027] The following embodiments may represent embodiments of a system for segmenting an object in image data, but may also represent embodiments of a method for segmenting an object in image data. In such embodiments, the method may include one or more additional steps corresponding to actions performed by the system.

[0028] Optionally, the processor subsystem is configured to:

[0029] Obtain a probability map of the image data, for example by applying a classification technique to the image data to obtain the probability map, wherein the probability map shows the probability that a corresponding image element of the image data belongs to the object;

[0030] For a corresponding segmentation result, identify the boundary of the object by using the probability map and using a corresponding probability threshold, thereby obtaining the first segmentation result and the second segmentation result of the object.

[0031] Thus, these two segmentation results are derived from a single probability map by using different probability thresholds.

[0032] Optionally, the processor subsystem is configured to obtain the segmentation result of the object by:

[0033] Apply the probability threshold to the probability map to obtain an object mask;

[0034] Determine the boundary of the object in the object mask to obtain the segmentation result, or use the object mask as the segmentation result of the object.

[0035] By applying a probability threshold to a probability map, a binary-like object mask can be obtained, where the boundaries of the object in the object mask can be determined (e.g., using edge detection techniques or using contour finding techniques, etc.), and the segmentation result of the object can be easily obtained. In some embodiments, the object mask can be directly used as the segmentation result of the object because the edges in the object mask may already clearly indicate the boundaries of the object.

[0036] Optionally, the processor subsystem is configured to generate the display data to show the visual representation of the first segmentation result and the visual representation of the second segmentation result on the display simultaneously or successively. When visualizing these two segmentation results simultaneously or successively, the differences in the segmentation results can be easily perceived. In some examples, the visual representation of the first segmentation result and the visual representation of the second segmentation result are shown as corresponding overlays on the image data. This facilitates the interpretation of the segmentation results because they are directly shown on the image data. In some other examples, the overlap between the segmentation results can be highlighted. This can further facilitate the interpretation of the segmentation results because the user can easily perceive where the two segmentations correspond and where they do not correspond.

[0037] Optionally, at least one of the first probability threshold and the second probability threshold is adjustable by the user. By allowing the user to adjust at least one of these two probability thresholds, the user can interactively determine how the segmentation result changes as a function of the probability threshold, which can provide the user with further qualitative feedback on the segmentation result.

[0038] Optionally, the first probability threshold is predetermined or selected by an algorithm, and the second probability threshold is adjustable by the user. The predetermined or automatically selected probability threshold can provide a reference threshold, and the probability threshold adjustable by the user can be compared with this reference threshold. For example, the predetermined or automatically selected probability threshold can provide a reference segmentation result, e.g., the best segmentation result according to a cost function. The user can then adjust the other probability threshold to determine how the segmentation result compares to the reference segmentation to obtain an indication of how certain the classification technique is about the object boundaries in the reference segmentation result.

[0039] Optionally, the system further includes a user input interface for receiving user input data from a user-operable user input device, where the processor subsystem is configured to generate the display data to show a graphical user interface to the user, and the graphical user interface includes graphical user interface elements that enable the user to adjust the probability threshold, and the graphical user interface elements are in the form of, for example, sliders. Sliders are an intuitive way to change the probability threshold.

[0040] Optionally, the processor subsystem is configured to: in response to the user's adjustment of the probability threshold, obtain an adjusted segmentation result of the object by reapplying the adjusted probability threshold to the probability map without recalculating the probability map. Calculating the probability map through classification techniques can be computationally complex. However, when the probability threshold is changed, the corresponding segmentation result of the object can be obtained without recalculating the probability map. Instead, the probability map obtained through classification techniques can be saved after calculation, and this probability map can be used together with each adjusted probability threshold to calculate a new object map. Calculating the object map is computationally less complex (much less) than calculating the probability map because it mainly involves thresholding operations.

[0041] Optionally, the probability threshold can be saved, such as a probability threshold selected by an algorithm or obtained by user adjustment, for example, saving the probability threshold as a bookmark. In some examples, the user can select the saved probability threshold again via a graphical user interface. In some examples, the saved probability threshold can be used as a default value, for example, for the next segmentation.

[0042] Optionally, the processor subsystem is configured to: precompute a series of segmentation results of the object using a series of probability thresholds, and allow the probability threshold to be adjusted by enabling the user to select a probability threshold from the series of probability thresholds. By precomputing the segmentation results, the user can quickly see the results of the adjusted probability threshold.

[0043] Optionally, the first probability threshold is selected by an algorithm such that a cost function quantifying the segmentation quality is at a maximum value, and wherein the second probability threshold is selected by an algorithm such that the cost function is at a minimum value. The cost function can be used to determine the quality of the segmentation result algorithmically. By selecting one probability threshold to obtain an optimal segmentation result according to the cost function and selecting another probability threshold to obtain the worst segmentation result according to the cost function, the difference between these two segmentation results can be maximized, which can make it easier for the user to determine the (in)determinacy of the classification technique for the object boundary. For example, if the object boundary does not shift or only shifts slightly between these two segmentation results despite the difference between the segmentation results being maximized, then this can indicate that the classification technique is relatively certain about the object boundary.

[0044] Optionally, the processor subsystem is configured to: obtain a series of segmentation results of the object using a corresponding series of probability thresholds, and generate the display data to show a visual representation of the series of segmentation results, such as displaying the visual representation of the series of segmentation results as an animation. It may be valuable for the user to be able to compare more than two segmentation results of an object obtained using different probability thresholds. By visualizing such a series (e.g., as an animation), the user can obtain a better indication of how certain the classification technique is about the object boundaries in the segmentation results.

[0045] Optionally, the processor subsystem is configured to obtain a first segmentation result and a second segmentation result of two or more objects, for example, from the same image or different images. And display the visual representation of the first segmentation result and the visual representation of the second segmentation result of two or more objects simultaneously or successively. In this way, the user (e.g., a radiologist) can easily verify whether the selected probability threshold is acceptable for both of these two objects.

[0046] Optionally, the processor subsystem is further configured to extract features from the segmentation results of the object to obtain feature values, where extracting the features includes: obtaining a first feature value from the first segmentation result of the object, and obtaining a second feature value from the second segmentation result of the object, and where generating the display data further includes: showing the visual representation of the first feature value together with the first segmentation result, and showing the visual representation of the second feature value together with the second segmentation result. Such features may also be referred to as "secondary" features, while the primary feature is the segmentation result itself; or such features may also be referred to as "metrics". Examples of secondary features (or "metrics") include, but are not limited to, the size of the object, the orientation of the object, the smoothness of the object boundary, etc. Generally speaking, secondary features can be a characterization (e.g., a numerical characterization) of the object. By visualizing the secondary features of these two segmentations in addition to the segmentation results themselves, the user can obtain a better indication of how certain the classification technique is about the object boundaries in the segmentation results. For example, if the value of the secondary feature "size" varies greatly between the segmentation results, this may indicate that the transition region is too wide, meaning that the classification technique is less certain about the exact boundaries of the object. Examples of visualizing the values of secondary features include bars, lines, text, etc.

[0047] Optionally, the image data is 2D image data, and the processor subsystem is configured to obtain a segmentation result as a 2D segmentation result of the object.

[0048] Optionally, the image data is 3D image data, and the processor subsystem is configured to obtain a segmentation result as a 3D segmentation result of the object, for example, using a 3D mesh.

[0049] In a further aspect of the present invention, there is provided a system for segmenting an object in image data, comprising:

[0050] an input interface for accessing the image data;

[0051] a processor subsystem configured to:

[0052] apply a classification technique to the image data to obtain a probability map of the image data, wherein the probability map shows the probability that a corresponding image element of the image data belongs to the object;

[0053] identify the boundary of the object by using the probability map and a probability threshold to obtain a segmentation result of the object, wherein the probability threshold indicates whether the image element belongs to the object when applied to the probability of the corresponding image element, and wherein obtaining the segmentation result includes: using a first probability threshold to obtain a first segmentation result of the object, and using a second probability threshold to obtain a second segmentation result of the object; and

[0054] generate display data including a visual representation of the first segmentation result and a visual representation of the second segmentation result; and

[0055] a display output interface for displaying the display data on a display.

[0056] In a further aspect of the present invention, there is provided a computer-implemented method for segmenting an object in image data, comprising:

[0057] accessing the image data;

[0058] applying a classification technique to the image data to obtain a probability map of the image data, wherein the probability map shows the probability that a corresponding image element of the image data belongs to the object;

[0059] identifying the boundary of the object by using the probability map and a probability threshold to obtain a segmentation result of the object, wherein the probability threshold indicates whether the image element belongs to the object when applied to the probability of the corresponding image element, and wherein obtaining the segmentation result includes: using a first probability threshold to obtain a first segmentation result of the object, and using a second probability threshold to obtain a second segmentation result of the object; and

[0060] generating display data including a visual representation of the first segmentation result and a visual representation of the second segmentation result; and

[0061] Output the display data for display on a display device.

[0062] Those skilled in the art will understand that two or more of the above embodiments, implementations, and / or optional aspects of the present invention can be combined in any useful way.

[0063] Based on this specification, those skilled in the art can make modifications and variations to the system, computer-implemented method, and / or computer program product, which correspond to the modifications and variations of another described entity. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] These and other aspects of the present invention will become apparent by reference to the embodiments, and will be further elaborated with reference to these embodiments. In the following description, these embodiments are described by way of example and with reference to the accompanying drawings, in which:

[0065] Figure 1 A system for segmenting an object in image data is shown, the system including a user interface subsystem that enables a user to interact with the system;

[0066] Figure 2A Illustrates how the segmentation result of a tumor varies as a function of the probability threshold used during segmentation, where the segmentation result and size of the tumor are visualized, and the probability threshold can be adjusted by the user via a slider, and the user can select the allowed uncertainty via the slider to thereby select the probability threshold;

[0067] Figure 2B Similar to Figure 2A , except that a second segmentation result obtained using a reference probability threshold is additionally shown, and the two segmentation results are shown on the screen simultaneously, overlapping one another, one on top of the other;

[0068] Figure 2C and Figure 2D are respectively similar to Figure 2A and Figure 2B , except that the segmentation result is a segmentation result based on a 3D mesh rather than a 2D contour segmentation result;

[0069] Figure 3A and Figure 3B show screenshots of an exemplary graphical user interface that allows a user to adjust the probability threshold for tumor segmentation;

[0070] Figure 4 Shows a method for segmenting an object in image data; and

[0071] Figure 5 Shows a non-transitory computer-readable medium including data.

[0072] It should be noted that these drawings are purely schematic and are not drawn to scale. In the drawings, elements corresponding to those already described may have the same reference numerals.

[0073] List of Reference Numerals

[0074] The following list of reference numerals is provided to facilitate the interpretation of the drawings, but the list of reference numerals should not be construed as limiting the claims.

[0075] 40 Data storage device

[0076] 50 Image data

[0077] 52 Segmentation data

[0078] 60 Display

[0079] 62 Display data

[0080] 80 (One or more) user input devices

[0081] 82 User input data

[0082] 100 System for segmenting objects in image data

[0083] 120 Data storage interface

[0084] 140 Processor subsystem

[0085] 142 - 146 Data communication

[0086] 160 Memory

[0087] 180 User interface subsystem

[0088] 182 Display output interface

[0089] 184 User input interface

[0090] 200 Slider for selecting (un)certainty

[0091] 210, 212 Object segmentation results based on 2D contours

[0092] 214, 216 Object segmentation results based on 3D meshes

[0093] 220 Deriving secondary features from segmentation results

[0094] 300, 302 Screenshots of the segmentation user interface

[0095] 310 Slider for selecting (un)certainty

[0096] Segmentation results of tumors 320 and 322

[0097] 400 Method for segmenting an object in image data

[0098] 410 Access image data

[0099] 420 Apply classification techniques to the image data

[0100] 430 Obtain at least two segmentation results of the object

[0101] 440 Generate display data showing two segmentation results

[0102] 450 Output the display data to a display

[0103] 500 Non - transient computer - readable medium

[0104] 510 Data representing a computer program Detailed description

[0105] Figure 1 A system 100 for segmenting an object in image data is shown. System 100 is shown as including a data storage interface 120 to a data storage device 40. The data storage device 40 may include image data 50 to be segmented. For example, the image data 50 may be medical image data (e.g., 2D image or 3D image), and may include (since it depicts) an anatomical structure or a part of an anatomical structure to be segmented by system 100. The data storage device 40 may also store other types of data (e.g., segmentation data 52 representing the result of segmentation or model data representing a machine - learning model ( Figure 1 not shown in the figure)). Generally, the data storage device 40 may be used as a short - term storage device and / or a long - term data storage device. In Figure 1 the example, the data storage interface 120 is shown as connected to an external data storage device 40 (e.g., a network - accessible data storage device 40). Alternatively, the data storage device 40 may be an internal data storage device of system 100 ( Figure 1 not shown in the figure). Generally, the data storage interface 120 may take various forms, such as a hard - disk or solid - state - disk interface to one or more hard disks and / or solid - state disks, or a network interface to a local area network (LAN) or wide area network (WAN). Generally, the data storage interface 120 may represent an example of an input interface as described elsewhere in this specification.

[0106] System 100 is also shown as including a processor subsystem 140 that is configured to communicate internally with the data storage interface 120 via data communication 142, communicate internally with the memory 160 via data communication 144, and communicate internally with the user interface subsystem 180 via data communication 146. The memory 160 can be, for example, a volatile memory in which a computer program can be loaded that can cause the processor subsystem 140 to perform the functions described in this specification (e.g., functions related to segmenting objects in image data using a probability threshold).

[0107] In some embodiments, system 100 can include a user interface subsystem 180 that can be configured to enable a user to interact with system 100, for example, using a graphical user interface, during operation of system 100. In particular, as also described elsewhere, the graphical user interface can enable the user to obtain visual feedback of the segmentation and adjust parameters of the segmentation (e.g., the probability threshold used in the segmentation as explained elsewhere in this specification). For this and other purposes, the user interface subsystem 180 is shown as including a user input interface 184 that is configured to receive user input data 82 from one or more user input devices 80 that can be operated by the user. The user input devices 80 can take various forms, including but not limited to a keyboard, a mouse, a touch screen, a microphone, etc. Figure 1 The user input devices are shown as a keyboard and a mouse 80. Generally, the user input interface 184 can be of a type corresponding to the type of the (one or more) user input devices 80, i.e., the user input interface 184 can be of the type of the corresponding user device interface. The user interface subsystem 180 is also shown as including a display output interface 182 that is configured to provide display data 62 to a display 60 to visualize the output of system 100. In Figure 1 the example, the display is an external display 60. Alternatively, the display can be an internal display of system 100 ( Figure 1 not shown).

[0108] In some embodiments, the processor subsystem 140 may be configured to apply classification techniques to the image data 50 during operation of the system 100 to obtain a probability map of the image data. The probability map may show the probability that a corresponding image element of the image data belongs to an object. The processor subsystem 140 may also be configured to obtain a segmentation result of the object by using the probability map and a probability threshold, where the probability threshold indicates whether the image element belongs to the object when applied to the probability of the corresponding image element, and where obtaining the segmentation result includes: obtaining a first segmentation result of the object using a first probability threshold, and obtaining a second segmentation result of the object using a second probability threshold. The processor subsystem 140 may also be configured to generate display data 62 for display on the display 60. The display data 62 may include a visual representation of the first segmentation result and a visual representation of the second segmentation result. Here and elsewhere, the term "visual" may refer to a visually perceivable representation without other limitations (e.g., without limiting the form of visualization).

[0109] Reference will be made Figure 2A to the following to more fully explain these and other operations of the system 100 and their various optional aspects.

[0110] In general, the system 100 may be embodied as a single device or apparatus, or embodied in a single device or apparatus. The device or apparatus may be a general-purpose device or apparatus (e.g., a workstation or a computer), but may also be a special-purpose device or apparatus (e.g., a patient monitor). The device or apparatus may include one or more microprocessors, which may represent the processor subsystem and may run appropriate software. The software may have been downloaded and / or stored in a corresponding memory (e.g., a volatile memory such as RAM or a non-volatile memory such as flash memory). Alternatively, the functional units of the system (e.g., the input interface, the user interface subsystem, and the processor subsystem) may be implemented in the device or apparatus in the form of programmable logic units (e.g., implemented as a field-programmable gate array (FPGA)). In general, each functional unit of the system 100 may be implemented in the form of a circuit. Note that the system 100 may also be implemented in a distributed manner (e.g., involving different devices or apparatuses). For example, the distribution may be according to a client-server model (e.g., using a server and workstations). For example, the user input interface and the display output interface may be part of a workstation, while the processor subsystem may be a subsystem of a server. Note that various other distributions are also conceivable.

[0111] The following describes Figure 1An embodiment of system 100, where the system is used to segment medical image data (e.g., image data obtained by an imaging modality (e.g., MRI, CT, X-ray, ultrasound, etc.)). However, it should be understood that the following also applies to the segmentation of other types of image data and objects. Non-limiting examples of such other types of image data include, but are not limited to, (3D) camera data, satellite image data, geographical information representable as an image, etc. As explained elsewhere, when using probability classification techniques (especially probability AI-based classification techniques) to segment medical images, it is relevant to provide the user with an indication of the (un)certainty of the likelihood that the segmentation boundary of the expected object matches the actual boundary of the object. For this purpose, Figure 1 System 100 can visualize one or more direct results of the uncertainty of the (one or more) segmentation results obtained by the system.

[0112] This can be further explained with reference to Figure 2A as follows. Figure 2A shows a slider 200 that can be presented to the user by the system via a graphical user interface. By operating the slider 200, the probability threshold used in the segmentation can be adjusted. In Figure 2A the leftmost example, the slider 200 can be set to 0% on a scale related to the uncertainty of the classification. In this example, the slider 200 is set such that an uncertainty of 0% is acceptable. This can correspond to a probability threshold of 100%, which means that only image elements classified as belonging to the object with a probability of 100% will be truly marked as "object" in the object map. In the basic example, each probability in the probability map can be compared with the probability threshold, and if the probability is equal to or greater than the probability threshold, the corresponding image element in the object mask can be set to "1", otherwise it can be set to "0". It should be understood that the object map can also take another form. For example, the object mask can include labels indicating which object the image element belongs to. Continuing to refer to Figure 2A , as a result of the thresholding, an object mask can be obtained, from which the segmentation result 210 of the object can be obtained, for example, in a manner known per se in the field of image segmentation (e.g., by edge detection, contour finding, binary erosion, etc.) to obtain the segmentation result 210 of the object. Figure 2AAlso shown is a secondary feature 220 being visualized, which can be derived from the segmentation result 210. In this example, the secondary feature 220 is the size of the object (in this example, "2 cm"). The user can adjust the slider 200 (e.g., by setting the slider 200 to 10%, 20%, or 30%, which can correspond to probability thresholds of 90%, 80%, and 70% respectively). By accepting a greater uncertainty, the object will appear larger in the corresponding segmentation result, and this larger size will also be shown by the secondary feature (e.g., "2.4 cm", "3.5 cm", "4.2 cm"). In other words, the segmentation result can shrink or grow according to the position of the slider 200.

[0113] In Figure 2A 's example, different segmentation results can be shown successively because if the slider is adjusted, the previous segmentation result will be replaced by the current segmentation result corresponding to the current position of the slider. Alternatively, multiple segmentation results can be shown simultaneously, e.g., adjacent to or overlapping each other. Although not shown in Figure 2A , the user can also select a segmentation result for further use, e.g., as the "final" segmentation result. In this way, the user can control how much uncertainty they accept and can accordingly adjust the segmentation result and optionally the secondary feature.

[0114] Figure 2B Shows an example similar to the example in Figure 2A , except that here two segmentation results are shown simultaneously, i.e., a reference segmentation result obtained using a reference probability threshold and another segmentation result obtained using a probability threshold selected by the user. The probability threshold can be selected by the slider 200. The reference segmentation result 210 can correspond to, for example, 0% uncertainty (100% probability threshold), and the reference segmentation result 210 and the other segmentation result 212 corresponding to the probability threshold selected by the user can be shown simultaneously as black outlines. Although not explicitly shown in Figure 2B , these two probability thresholds can also be adjusted.

[0115] Figure 2C And Figure 2D Are respectively similar to Figure 2A And Figure 2B , except that the segmentation results are segmentation results based on a 3D mesh rather than a 2D contour. Therefore, Figure 2C Shows a 3D mesh 214 depicting the boundary of the object in the 3D image data, while Figure 2D Shows the 3D mesh 214 as the reference segmentation result and simultaneously shows a 3D mesh 216 obtained using the probability threshold selected by the user.

[0116] Figure 3A and Figure 3B shows screenshots of exemplary graphical user interfaces 300, 302 that allow a user to adjust a probability threshold for tumor segmentation. In this graphical user interface, a slider 310 is shown at the top, and the slider 310 enables the user to control an uncertainty threshold. Directly below the slider is the rendering of 3D meshes 320, 322 (left side) and 2D segmentation contours (right side) superimposed on CT slices. The colors in the mesh ( Figure 3A , Figure 3B not shown in) can represent different structures. For example, yellow can refer to a tumor, blue can refer to blood vessels, pink can refer to the contact area between the tumor and blood vessels, etc. It is worth noting that the contact area (also known as tumor involvement) is crucial for assessing the resectability of a tumor. Below these 2D and 3D images is the visualization of secondary features derived from the contact area (e.g., the tumor involvement angle for each slice (which is the angle formed by the contact area relative to the center point of the blood vessel), and the length of tumor involvement across slices). The slider 310 can control the uncertainty threshold for tumor segmentation. When comparing Figure 3A with Figure 3B , it can be seen that when adjusting the uncertainty threshold, the results can be directly observed because the size of the segmentation results 320, 322 and the depicted secondary features change.

[0117] It should be understood that the probability threshold adjustable by the user can be implemented in various ways. For example, the user can directly adjust the probability threshold, or indirectly adjust the probability threshold (e.g., by adjusting the (un)certainty threshold). For such adjustment, graphical user elements (e.g., sliders) can be used. Alternatively, any other type of user input (e.g., pressing a button, mouse click) can be used. The adjustment of the probability threshold can also be (partially) automated (e.g., in the form of an animation). Through such an animation, different segmentation results can be successively shown to the user.

[0118] In addition, after adjusting the probability threshold, the segmentation result and optional secondary features can change according to the adjusted probability threshold. The information required for such a change may have been pre-computed so that when the probability threshold is adjusted, only the visibility of the corresponding information changes. For example, if the probability threshold is set to 80%, the segmentation result and secondary features corresponding to 80% can be selected for visualization, and other segmentation results and secondary features can be hidden. This can be implemented by the system pre-computing the segmentation results and secondary features for a range of probability thresholds and allowing the user to adjust the probability threshold by selecting a threshold from this range of probability thresholds. In other examples, a probability map may be pre-computed, but the object map can be re-computed after adjusting the probability threshold (e.g., by applying the adjusted probability threshold to the probability map).

[0119] Continuing to refer to the probability map and the object map, note the following. The classification technique can be a probabilistic AI classification technique (e.g., a Bayesian model, using Monte Carlo dropout). Such a technique can produce a probability map. From this map, the average label value for each image element (e.g., pixel or voxel) can be derived. For example, the label can be: 1 for tumor and 0 for background. An average label value of 0.6 can indicate a 60% probability that the image element is part of a tumor. Then, the probability map can be thresholded according to the selected probability threshold (e.g., by comparing each probability with the probability threshold).

[0120] It should also be noted that the object mask can be a binary mask. If the mask is a 3D mask for 3D image data, the mask can be converted to a mesh (e.g., using the marching cubes algorithm). If the mask is a 2D mask for 2D image data, the boundaries in the object mask can be detected in various ways (e.g., using edge detection or binary erosion).

[0121] In general, the mapping from uncertainty values to probability values (e.g., the mapping from uncertainty values selected by a slider to probability thresholds) can be linear because 0% uncertainty can correspond to a 100% probability threshold, 50% uncertainty can correspond to a 50% probability threshold, and 100% uncertainty can correspond to a 0% probability threshold. However, the mapping can also be non-linear, for example, taking into account differences in the interpretation of uncertainty between the user and the classification technique. Additionally, although in some embodiments the probability threshold is shown as being used in a basic thresholding operation, the probability threshold can also be used in more complex functions (e.g., functions that take into account surrounding probabilities) or as a relative threshold (e.g., relative to other local probabilities) rather than an absolute threshold.

[0122] Figure 4A block diagram of a computer-implemented method 400 for segmenting objects in image data is shown. Method 400 may correspond to Figure 1 the operation of system 100. However, this is not a limitation, as another system, apparatus, or device may also be used to perform computer-implemented method 400.

[0123] Method 400 is shown to include: in an operation titled "Access Image Data", accessing 410 image data as described elsewhere in this specification; and in an operation titled "Apply Classification Techniques to Image Data", applying 420 classification techniques to the image data to obtain a probability map of the image data, where the probability map shows the probability that a corresponding image element of the image data belongs to an object; in an operation titled "Obtain at Least Two Segmentation Results of the Object", obtaining 430 a segmentation result of the object by using the probability map and using a probability threshold to identify the boundary of the object, where the probability threshold indicates whether the image element belongs to the object when applied to the probability of the corresponding image element, and where obtaining the segmentation result includes: using a first probability threshold to obtain a first segmentation result of the object, and using a second probability threshold to obtain a second segmentation result of the object; in an operation titled "Generate Display Data Showing These Two Segmentation Results", generating 440 display data that includes a visual representation of the first segmentation result and a visual representation of the second segmentation result; and in an operation titled "Output the Display Data to a Display", outputting 450 the display data for display on a display.

[0124] It should be understood that, generally speaking, Figure 4 the operations of method 400 may be performed in any suitable order (e.g., sequentially, simultaneously, or a combination thereof), and in cases where applicable, a specific order is required (e.g., according to input / output relationships). These operations may also be performed as part of other operations.

[0125] The method may be implemented on a computer as a computer-implemented method, implemented as dedicated hardware, or implemented as a combination of the two. Similarly, as Figure 5 shown, instructions for a computer (e.g., executable code) may be stored on a computer-readable medium 500, e.g., stored in the form of a series of machine-readable physical markers 510 and / or stored as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code may be stored in a transient or non-transient manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 5 A memory device 500 is shown.

[0126] Examples, embodiments or optional features, whether or not indicated as non-limiting, should not be construed as limiting the invention claimed.

[0127] It should be noted that the above embodiments illustrate rather than limit the invention, and those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs in parentheses shall not be construed as limiting the claim. The use of the verb "comprise" and its conjugations does not exclude the presence of elements or steps other than those recited in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one of..." when preceding a list of elements or a group of elements denote the selection of all elements or any subset of elements from the list of elements or the group of elements. For example, the expression "at least one of A, B and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B and C. The invention may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a device claim enumerating several units, several of these units may be implemented by the same hardware. The fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A system (100) for segmenting an object in image data, comprising: an input interface (120) for accessing the image data (50); a processor subsystem (140) configured to: obtain a segmentation result (210 - 216, 320, 322) of the object, wherein the segmentation result includes image elements in the image data that may be part of the object, and the image elements that may be part of the object are determined by a probability value of a corresponding image element exceeding a probability threshold, and wherein obtaining the segmentation result includes: obtaining a first segmentation result (320) of the object using a first probability threshold, and obtaining a second segmentation result (322) of the object using a second probability threshold, wherein the first probability threshold is different from the second probability threshold; and generate display data (62), the display data including a visual representation of the first segmentation result and a visual representation of the second segmentation result; and a display output interface (182) for displaying the display data on a display.

2. The system (100) according to claim 1, wherein, the processor subsystem (140) is configured to: obtain a probability map of the image data, for example, by applying a classification technique to the image data to obtain the probability map of the image data, wherein the probability map shows the probability that a corresponding image element of the image data belongs to the object; for a corresponding segmentation result, identify the boundary of the object by using the probability map and using a corresponding probability threshold, thereby obtaining the first segmentation result and the second segmentation result (210 - 216, 320, 322) of the object.

3. The system (100) according to claim 2, wherein, the processor subsystem (140) is configured to obtain the segmentation result of the object by: applying the probability threshold to the probability map to obtain an object mask; determining the boundary of the object in the object mask.

4. The system (100) according to any one of claims 1 to 3, wherein, the processor subsystem (140) is configured to generate the display data (62) to simultaneously or successively show the visual representation of the first segmentation result (320) and the visual representation of the second segmentation result (322) on the display.

5. The system (100) according to claim 4, wherein, the visual representation of the first segmentation result and the visual representation of the second segmentation result are shown as corresponding overlays on the image data (50).

6. The system (100) according to any one of claims 1 to 5, wherein, at least one of the first probability threshold and the second probability threshold is adjustable by a user.

7. The system (100) according to any one of claims 1 to 6, wherein, the first probability threshold is predetermined or selected by an algorithm, and the second probability threshold is adjustable by a user.

8. The system (100) according to claim 6 or 7, wherein, the system further includes a user input interface (180) for receiving user input data (82) from a user-operable user input device (80), wherein the processor subsystem (140) is configured to generate the display data (62) to show a graphical user interface to the user, wherein the graphical user interface includes graphical user interface elements that enable the user to adjust the probability threshold, and the graphical user interface elements are in the form of, for example, sliders (200, 310).

9. The system (100) according to any one of claims 6 to 8, which depends on claim 2, wherein, the processor subsystem (140) is configured to: in response to the user's adjustment of the probability threshold, obtain an adjusted segmentation result of the object by reapplying the adjusted probability threshold to the probability map without recalculating the probability map.

10. The system (100) according to any one of claims 6 to 9, wherein, the processor subsystem (140) is configured to: pre-compute a series of segmentation results of the object using a series of probability thresholds, and allow the probability threshold to be adjusted by enabling the user to select a probability threshold from the series of probability thresholds.

11. The system (100) according to any one of claims 1 to 10, wherein, the first probability threshold is selected by an algorithm such that a cost function quantifying the segmentation quality is at a maximum value, and wherein the second probability threshold is selected by an algorithm such that the cost function is at a minimum value.

12. The system (100) according to any one of the above claims, wherein, the processor subsystem (140) is configured to: obtain a series of segmentation results of the object using a corresponding series of probability thresholds, and generate the display data to show a visual representation of the series of segmentation results, for example, display the visual representation of the series of segmentation results as an animation.

13. The system (100) according to any one of claims 1 to 12, wherein, the processor subsystem (140) is further configured to extract features from the segmentation results of the object to obtain feature values, wherein extracting the features includes: obtaining a first feature value from the first segmentation result of the object, and obtaining a second feature value from the second segmentation result of the object, and wherein generating the display data (62) further includes: showing a visual representation of the first feature value together with the first segmentation result, and showing a visual representation of the second feature value together with the second segmentation result.

14. A computer-implemented method (400) for segmenting an object in image data, comprising: accessing (410) the image data; Obtain (420, 430) a segmentation result of the object, wherein the segmentation result includes image elements in the image data that may be part of the object, and the image elements that may be part of the object are determined by the probability value of the corresponding image element exceeding a probability threshold. Obtaining the segmentation result includes: using a first probability threshold to obtain a first segmentation result of the object, and using a second probability threshold to obtain a second segmentation result of the object, wherein the first probability threshold is different from the second probability threshold; Generate (440) display data, the display data including a visual representation of the first segmentation result and a visual representation of the second segmentation result; and Output (450) the display data for display on a display.

15. A transient or non-transient computer-readable medium (500) comprising data (510) representing a computer program, the computer program including instructions for causing a processor system to perform the method according to claim 14.

Images