Image segmentation confidence determination

Abstract

Examples for determining a confidence level associated with image segmentation are disclosed. The confidence level associated with a collective image segmentation result can be determined by generating multiple individual segmentation results from the same image data, respectively. The examples can then aggregate the individual segmentation results to form a collective image segmentation result, and measure the spread of each individual segmentation result from the collective image segmentation result. The measured spread of each individual segmentation result can then be used to determine a confidence level associated with the collective image segmentation result. This can allow the confidence level associated with the collective image segmentation result to be determined. The confidence level can be determined without comparing a ground truth to the collective image segmentation result.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 581,577, filed on September 24, 2019, and claims the benefit of U.S. Provisional Application No. 62 / 869,793, filed on July 2, 2019, the contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to image segmentation, and more particularly to devices, systems, and methods for determining, and in some cases indicating, a confidence level associated with image segmentation.Examples disclosed herein may be applied, for example, to medical imaging. Background Art

[0004] Medical imaging techniques can be used to collect image data and generate in vivo visualizations of anatomical regions of interest. One such example is intravascular imaging, where vascular structures and lumens can be imaged. For example, intravascular imaging can be used to generate one or more images of the lumen of a coronary artery, the morphology of the coronary artery wall, and devices such as stents at or near the coronary artery wall. These images can be used for diagnostic purposes, such as to diagnostically identify important features of the blood vessel.

[0005] However, the information collected during medical imaging may include a large amount of data, some of which may be irrelevant to the purpose for which the imaging was performed. Therefore, the image data collected during imaging is further analyzed to simplify image interpretation and automatic identification of certain anatomical features and / or measurement of clinically useful parameters.

[0006] As an example, image segmentation can be used to simplify the interpretation and automated identification and / or measurement of key features in image data. Image segmentation can be used to divide an image or a group of images into one or more regions of interest to facilitate identification of relevant information in the image. Summary of the invention

[0007] Described herein are illustrative examples for determining confidence levels, such as error probabilities, associated with image segmentation results. Because in many applications, the results of image segmentation can be used to facilitate meaningful decision making, the reliability of the segmentation results is important. In the example of medical imaging, the results of image segmentation can be used to facilitate the identification of clinically relevant information, and therefore, the reliability of the segmentation results may be important for accurately identifying clinically relevant information. Therefore, the examples disclosed herein can estimate and provide confidence levels for segmentation results, so that users can consider the estimated confidence levels when using the segmentation results.

[0008] Estimation of the confidence level of the segmentation results can be particularly helpful in autonomous segmentation. Manual segmentation of image data by a user can be tedious and time consuming, and thus autonomous segmentation that performs segmentation operations on image data without or with minimal user input during execution may be desirable. However, autonomous segmentation may introduce a degree of uncertainty. Providing an estimated confidence level associated with an automatic segmentation result can allow a user to be more informed when relying on the automatic segmentation result.

[0009] It is noteworthy that the examples disclosed herein can determine the confidence level associated with the image segmentation result in applications where there is no ground truth to compare the segmentation result. In many applications, image data is generated and segmented to determine one or more unknown information. As such, because the unknown information can be determined by the segmentation result, the lack of known comparison points for the segmentation result can pose a challenge in determining the confidence level associated with the segmentation result. The examples disclosed herein can solve this problem by determining the confidence level associated with the segmentation result without comparing the ground truth to the segmentation result. For example, in intravascular imaging, image data of a vessel wall, a vessel lumen, and a plaque can be generated and segmented to determine a previously unknown boundary between the vessel wall and the vessel lumen. Because the boundary between the vessel wall and the lumen is unknown, the examples disclosed herein can provide such image data with a segmentation result including a confidence level associated with the segmentation boundary between the vessel wall and the lumen, without using a known boundary between the vessel wall and the lumen to determine the confidence level.

[0010] In some examples disclosed herein, a confidence level associated with an image segmentation result can be determined by generating multiple individual segmentation results from the same image data, respectively. These examples can then measure the spread of each individual segmentation result based on the consistency of all segmentation results (e.g., based on a collective segmentation result formed from the multiple individual segmentation results). The measured spread of each individual segmentation result can then be used to determine a confidence level associated with a consistent or collective image segmentation result presented to a user. This can allow the confidence level associated with a consistent or collective segmentation result to be determined without comparing the ground truth to the consistent or collective segmentation result.

[0011] In one example, a method includes receiving, by a processor, an image containing a cross-sectional view of an object. The method also includes, for each of a plurality of models, generating, by the processor and using the corresponding model, a boundary prediction of the object within the image, wherein each boundary prediction includes a plurality of boundary prediction portions and each of the plurality of models includes a unique model. The method also includes, by the processor and based on the boundary prediction of each of the plurality of models, generating a cluster boundary prediction of the object within the image, the cluster boundary prediction including a plurality of cluster boundary prediction portions, each cluster boundary prediction portion corresponding to one of the plurality of boundary prediction portions of each boundary prediction. The method also includes, for each of the plurality of cluster boundary prediction portions, determining, by the processor and based on a comparison between the corresponding cluster boundary prediction portion and a corresponding boundary prediction portion of the boundary prediction, a confidence level of the corresponding cluster boundary prediction portion. The method also includes, by the processor and for display on a display device, outputting a graphical indication of each of the plurality of cluster boundary prediction portions, wherein each cluster boundary prediction portion has a visual feature indicating a confidence level of the corresponding cluster boundary prediction portion.

[0012] In another example, a computing device includes at least one processor and a storage component configured to store instructions. The instructions are executable by the at least one processor to receive an image containing a cross-sectional view of an object. The instructions are also executable by the at least one processor to generate a boundary prediction of an object within the image using a corresponding model for each of a plurality of models, wherein each boundary prediction includes a plurality of boundary prediction portions, and each of the plurality of models includes a unique model. The instructions are also executable by the at least one processor to generate a cluster boundary prediction of the object within the image based on the boundary prediction of each of the plurality of models, the cluster boundary prediction including a plurality of cluster boundary prediction portions, each cluster boundary prediction portion corresponding to one of the plurality of boundary prediction portions of each boundary prediction. The instructions are also executable by the at least one processor to determine, for each of the plurality of cluster boundary prediction portions, a confidence level of the corresponding cluster boundary prediction portion based on a comparison between the corresponding cluster boundary prediction portion and a corresponding boundary prediction portion of the boundary prediction. The instructions are also executable by the at least one processor to output a graphical indication of each of the plurality of cluster boundary prediction portions for display on a display device, wherein each cluster boundary prediction portion has a visual feature indicating a confidence level of the corresponding cluster boundary prediction portion.

[0013] In another example, a non-transitory computer-readable storage medium storing instructions is described, which when executed causes one or more processors of a computing device to receive an image containing a cross-sectional view of an object. The instructions also cause the one or more processors to generate a boundary prediction of an object within the image using a corresponding model for each of a plurality of models, wherein each boundary prediction includes a plurality of boundary prediction portions, and each of the plurality of models includes a unique model. The instructions also cause the one or more processors to generate a cluster boundary prediction of the object within the image based on the boundary prediction of each of the plurality of models, the cluster boundary prediction including a plurality of cluster boundary prediction portions, each cluster boundary prediction portion corresponding to one of the plurality of boundary prediction portions of each boundary prediction. The instructions also cause the one or more processors to determine, for each of the plurality of cluster boundary prediction portions, a confidence level of the corresponding cluster boundary prediction portion based on a comparison between the corresponding cluster boundary prediction portion and a corresponding boundary prediction portion of the boundary prediction. The instructions also cause the one or more processors to output a graphical indication of each of the plurality of cluster boundary prediction portions for display on a display device, wherein each cluster boundary prediction portion has a visual feature indicating a confidence level of the corresponding cluster boundary prediction portion.

[0014] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The following drawings illustrate specific examples of the present invention and therefore do not limit the scope of the present invention. The drawings are intended to be used in conjunction with the description in the following description. Examples of the present invention will be described below in conjunction with the drawings, wherein the same reference numerals represent the same elements.

[0016] Figure 1 is an example system that includes a camera configured to capture images and a computing device configured to analyze the images according to one or more techniques described herein.

[0017] Figure 2 is an example of a cross-sectional view of a blood vessel showing a plurality of exemplary individual segmentation results and an exemplary collective segmentation result formed from the plurality of individual segmentation results according to one or more techniques described herein.

[0018] Figure 3 yes Figure 2 An example of a cross-sectional view of a blood vessel in FIG. 1 , with exemplary collective segmentation results with associated confidence levels according to one or more techniques described herein.

[0019] Figure 4 is a diagram illustrating an example of a method of determining a confidence level using clustering of individual segmentation results in accordance with one or more techniques described herein. Figure 4The schematic diagram shows relatively strong clustering around specific values.

[0020] Figure 5 is another schematic diagram showing an example of a method of determining a confidence level using clustering of individual segmentation results. Figure 5 The schematic diagram shows the lack of strong clustering around any particular value.

[0021] Figure 6 is a flow chart illustrating an example process for predicting cluster boundaries of objects within an image in accordance with one or more techniques described herein. DETAILED DESCRIPTION

[0022] The following detailed description is exemplary in nature and provides some practical illustrations and examples. Those skilled in the art will recognize that many of the examples mentioned have a variety of suitable alternatives.

[0023] Examples disclosed herein can determine a confidence level associated with an image segmentation result. In particular, examples disclosed herein can use multiple individual segmentation results from the same image data. Such examples can apply the statistical properties of the integration process to estimate the confidence of the segmentation performance by measuring the spread of each individual segmentation result from the consistency of all segmentation results (e.g., the collective segmentation result formed from multiple individual segmentation results). This estimated confidence of the image segmentation result can provide users with additional information about how to use the segmentation result without knowing the basic facts.

[0024] Figure 1 is an example system that includes a camera configured to capture images and a computing device configured to analyze the images according to one or more techniques described herein. Figure 1 Only one particular example of computing device 2 is illustrated, and many other examples of computing device 2 may be used in other situations and may include a subset of the components included in the example computing device 2 or may include Figure 1 For example, the camera 20 may be incorporated into the computing device 2, or operably connected in a manner other than wireless connection, rather than being a separate component.

[0025] like Figure 1In the example shown, the computing device 2 includes one or more processors 4, one or more input / output (I / O) components 6, one or more communication units 14, and one or more storage components 8. The storage component 8 of the computing device 2 includes a clustering module 10 and a prediction model 12. For the purposes of this disclosure, the prediction model can be similar to a segmentation operation, where each model describes a different segmentation operation. A communication channel (not shown) can interconnect each of the components 4, 6, 8, 14 for inter-component communication (physically, communicatively, and / or operationally). In some examples, the communication channel can include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0026] One or more communication units 14 of computing device 2 may communicate with external devices via one or more wired and / or wireless networks by sending and / or receiving network signals over the one or more networks. Examples of communication unit 14 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples of communication unit 14 may include a shortwave radio, a cellular data radio, a wireless network radio, and a universal serial bus (USB) controller. Communication unit 14 may include a wireless interface configured to communicate with external devices in accordance with, for example, a wireless interface such as a wireless interface. or LTE wireless protocols.

[0027] One or more I / O components 6 may include input components that can receive input. Examples of input are tactile, audio, and video input. In one example, the I / O components 6 of the computing device 2 include an intravascular ultrasound catheter, a state-sensitive input device (e.g., a touch-sensitive screen, PSD), a mouse, a keyboard, a voice response system, a camera, a microphone, or any other type of device for detecting input from a person or a machine. In some examples, the I / O components 6 may include one or more sensor components, one or more location sensors (GPS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., accelerometers, gyroscopes), one or more pressure sensors (e.g., barometers), one or more ambient light sensors, and one or more other sensors (e.g., microphones, cameras, infrared proximity sensors, hygrometers, etc.).

[0028] I / O components 6 may also include one or more output components that generate output. Examples of output are tactile, audio, and video output. In one example, I / O components 6 of computing device 2 include a PSD, a sound card, a video graphics adapter card, speakers, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device for generating output to a person or machine.

[0029] One or more processors 4 may implement functions associated with computing device 2 and / or execute instructions associated with computing device 2. Examples of processor 4 include an application processor, a display controller, an auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, a processing unit, or a processing device. Modules 10 and models 12 may be operated by processor 4 to perform various actions, operations, or functions of computing device 2. For example, processor 4 of computing device 2 may retrieve and execute instructions stored by storage component 8 that cause processor 4 to perform operations of module 10 and model 12. When the instructions are executed by processor 4, the instructions may cause computing device 2 to store information in storage component 8.

[0030] One or more storage components 8 within computing device 2 may store information for processing during operation of computing device 2 (e.g., computing device 2 may store data accessed by module 10 and predictive model 12 during execution of computing device 2). In some examples, storage component 8 is a temporary memory, meaning that the primary purpose of storage component 8 is not long-term storage. Storage component 8 on computing device 2 may be configured to store information short-term as volatile memory and therefore does not retain stored content if power is removed. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art.

[0031] In some examples, the storage component 8 also includes one or more computer-readable storage media. In some examples, the storage component 8 includes one or more non-transitory computer-readable storage media. The storage component 8 can be configured to store a larger amount of information than the amount of information typically stored by volatile memory. The storage component 8 can also be configured to store information for a long time as a non-volatile memory space and retain the information after a power on / off cycle. Examples of non-volatile memory include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. The storage component 8 can store program instructions and / or information (e.g., data) associated with the module 10 and the prediction model 12. The storage component 8 may include a memory configured to store data or other information associated with the module 10 and the prediction model 12.

[0032] The camera 20 may be any image capture device capable of capturing an image of an object. In some cases, the camera 20 may be an intravascular ultrasound catheter configured to capture an image of a cross section of a coronary artery of, for example, the patient 70. However, in other cases, the camera 20 may be any other type of camera configured to capture a cross-sectional image of an object. Therefore, although described throughout the present disclosure as measuring the boundaries of an intravascular object after capturing an intravascular ultrasound image, the same techniques may be applied to any image where the boundaries of an object within the image need to be defined. For example, similar techniques may be applied to natural images (e.g., color images, grayscale images), images of different medical modalities (e.g., images captured by magnetic resonance imaging (MRI), X-rays, computed tomography (CT), positron emission tomography (PET), single photon emission computed tomography (SPECT), and ultrasound), and non-natural images (e.g., thermal images, ultraviolet images). In addition, although the techniques described herein are described for cross-sectional areas, the same techniques may be applied to objects within any image as long as the boundaries of such objects can be determined within the image.

[0033] According to the technology of the present disclosure, clustering module 10 can receive an image containing a cross-sectional view of an object. Clustering module 10 can receive the image from a camera such as camera 20. Camera 20 can be external to clustering module 10, where clustering module 10 receives the image via communication unit 14. In other cases, camera 20 can be incorporated into computing device 2. The image can be any image that includes an object with known or unknown boundaries within the image, including a cross-section of the object. For example, camera 20 can be an intravascular ultrasound device configured to capture an image of a vein or artery of patient 70.

[0034] For each of the multiple models, the clustering module 10 can generate a boundary prediction of an object in the image using the corresponding model. Boundary prediction can also be referred to as image segmentation or a separate segmentation result. Each of the multiple models can be a unique model. In addition, each boundary prediction can include multiple boundary prediction parts. For example, the multiple models can include any type of machine learning or deep learning model, pattern recognition model, computer vision model or parameter correction model, etc. For example, a support vector machine (SVM) model, a UNet model, a random forest model, and an edge detection computer vision model such as a Sobel filter, and their variations under different parameters can be included in the multiple models. Ultimately, the multiple models can include up to nine or more different models, each of which can be used to predict the boundary of an object in the image.

[0035] The clustering module 10 can generate cluster boundary predictions of objects within the image based on the boundary predictions of each model in the multiple models. The cluster boundary predictions can also be referred to as collective segmentation results. The clustering module 10 can use an integration technique together with a clustering metric to determine the cluster boundary predictions. In some cases, the clustering metric of the cluster boundary predictions can be the median of the independent image segmentation results (or the corresponding boundary prediction parts of the boundary predictions). In some cases, the clustering metric of the cluster boundary predictions can be an average boundary obtained by averaging each of the corresponding boundary prediction parts of the boundary predictions of the multiple models. In other cases, the cluster boundary predictions can be weighted cluster boundary predictions of objects within the image. For example, the clustering module 10 can determine the weight of the corresponding model for each of the multiple models. Then, the clustering module 10 can generate weighted cluster boundary predictions based on the corresponding boundary prediction parts of the boundary predictions and the weights of each model in the multiple models by applying the weights to each of the boundary predictions and averaging the corresponding boundary prediction parts of the boundary predictions in combination with the weights.

[0036] The cluster boundary prediction may include multiple cluster boundary prediction parts, each cluster boundary prediction part corresponding to one of the multiple boundary prediction parts of each boundary prediction. For example, if each boundary prediction is decomposed into 100 boundary prediction parts, the cluster boundary prediction can also be divided into 100 cluster boundary prediction parts, where the corresponding parts have the same position relative to the object in the image. For each of the multiple cluster boundary prediction parts of the cluster boundary prediction, the clustering module 10 can determine the confidence level of the corresponding cluster boundary prediction part based on the comparison between the corresponding cluster boundary prediction part and the corresponding boundary prediction part of the boundary prediction.

[0037] In some examples, when determining the confidence level, for each of the plurality of models, the clustering module 10 may determine the spread of the corresponding model based on a comparison between the cluster boundary prediction portion and the corresponding boundary prediction portion of the boundary prediction. One such example of a spread may be a z-score, although any measure of the deviation between two values ​​may be used for the purposes of the techniques described herein. The clustering module 10 may then determine the confidence level of the corresponding cluster boundary prediction portion based on the spread of each of the corresponding boundary prediction portions of the boundary prediction.

[0038] As defined by statistical properties, a z-score is equivalent to the number of standard deviations a particular value is from the mean. Clustering module 10 can determine the z-score of a corresponding model by first creating a prediction distribution using the cluster boundary prediction portion and the corresponding boundary prediction portion of the boundary prediction, such as Figure 4 and Figure 5 The prediction distribution of , where the cluster boundary prediction part is the average of the prediction distribution. This prediction distribution may be like the points along the line Figure 1The clustering module 10 may then determine the standard deviation of the predicted distribution. The standard deviation may be calculated according to typical mathematical rules, or by proprietary adjustments to typical algorithms. The clustering module 10 may then determine the z-score of each of the corresponding boundary prediction portions of the boundary prediction as the number of standard deviations that the respective corresponding boundary prediction portion is from the cluster boundary prediction portion.

[0039] In some cases, one or more of the models may produce outlier results: at least for the cluster boundary prediction portion of the cluster boundary prediction, the model's prediction is clearly wrong. In practice, the clustering module 10 can determine that the spread of the first corresponding boundary prediction portion of the first boundary prediction exceeds the deviation threshold of the prediction distribution, and the first boundary prediction is associated with the first model of the multiple models. In some examples, the deviation threshold can be a user-defined threshold, where other examples may include a default or predefined threshold. In response to making this determination, the clustering module 10 can remove the first model from the multiple models to generate a model subset that includes all other models from the multiple models except the first model (e.g., the model subset includes non-outlier cluster boundary prediction portions of other individual segmentation results). The clustering module 10 can then recalculate at least the corresponding cluster boundary prediction portion of the cluster boundary prediction using only the model subset, rather than allowing the erroneous model to change the results of the algorithm.

[0040] The clustering module 10 can output at least a presentation of the collective segmentation results for display on a display device (e.g., I / O component 6), the presentation including a graphical indication of each of the multiple cluster boundary prediction parts for cluster boundary prediction. Each of the multiple cluster boundary prediction parts has a visual feature or visual indicator indicating the confidence level of the corresponding cluster boundary prediction part. The visual feature can be any distinguishing visual feature that can indicate the confidence level of the relevant specific cluster boundary prediction part. Examples of visual features include line color, fill pattern, line thickness, line segmentation, text indication, graphic icon, or any combination thereof. For example, when the visual feature includes line color, a green indication of the cluster boundary prediction part for cluster boundary prediction can indicate a high confidence level of the cluster boundary prediction part of the cluster boundary prediction. In contrast, the cluster boundary prediction part of the cluster boundary prediction with a low confidence level can be depicted as red in the output, while the part of the cluster boundary prediction with a medium confidence level can be depicted as yellow. This may be a color spectrum, changing from a green spectrum to a red spectrum as the confidence level decreases.

[0041] Figure 2An example of a cross-sectional view of a blood vessel 100 is shown. The cross-sectional view of the blood vessel 100 includes a plurality of exemplary individual segmentation results 105A-105E and an exemplary collective segmentation result 110. The collective segmentation result 110 may be formed by one or more individual segmentation results 105A-105E. For example, one or more individual segmentation results 105A-105E may be aggregated to form the collective segmentation result 110. In the example shown, the collective segmentation result 110 is formed by each of the individual segmentation results 105A-105E. Although the example shown is described with reference to five individual segmentation results 105A-105E, other numbers of individual segmentation results (e.g., more than five, less than five) may be used in other examples within the scope of the present disclosure.

[0042] like Figure 2 As shown, the cross-sectional view of the blood vessel 100 includes a blood vessel wall 115 that defines a blood vessel lumen 120. Multiple independent segmentation operations are performed on the same image data. In this example, multiple independent segmentation operations are performed on image data corresponding to the cross-sectional view of the blood vessel 100. In the example shown, multiple independent segmentation operations (or models) are performed to identify / predict an interface or boundary between the blood vessel wall 115 and the blood vessel lumen 120.

[0043] Each of the plurality of independent segmentation operations may generate an independent segmentation result. In the illustrated example, each independent segmentation operation generates a corresponding independent segmentation result 105A-105E. For example, the first segmentation operation may generate a first independent segmentation result 105A, the second segmentation operation may generate a second independent segmentation result 105B, the third segmentation operation may generate a third independent segmentation result 105C, the fourth segmentation operation may generate a fourth independent segmentation result 105D, and the fifth segmentation operation may generate a fifth independent segmentation result 105E. Each segmentation operation may be different. For example, in the illustrated example, each segmentation operation may be a different segmentation algorithm for identifying the interface between the vessel wall 115 and the vessel lumen 120. Moreover, each of the different segmentation algorithms may be performed on the same image data corresponding to the cross-sectional view of the vessel 100. Therefore, when estimating the interface between the vessel wall 115 and the vessel lumen 120, each of the independent segmentation results 105A-105E generated by these different segmentation operations may include one or more regions having a location similar to that of other independent segmentation results, and one or more regions having a location significantly different from that of other independent segmentation results.

[0044] like Figure 2As shown, the cross-sectional view of the blood vessel 100 may include one or more regions 125A, 125B, 125C, where the individual segmentation results 105A-105E identify similar locations as interfaces between the blood vessel wall 115 and the blood vessel lumen 120. At the regions 125A, 125B, 125C, two or more of the individual segmentation results 105A-105E generally overlap, thus identifying similar locations as interfaces between the blood vessel wall 115 and the blood vessel lumen 120. For example, Figure 2 As shown, at each of regions 125A, 125B, 125C, two or more (e.g., each) of the individual segmentation results 105A-105E are within a predetermined range of each other and are therefore considered to be substantially overlapping and identify similar locations as interfaces between the vessel wall 115 and the vessel lumen 120.

[0045] Likewise Figure 2 As shown, the cross-sectional view of the blood vessel 100 may include one or more regions 130A, 130B, 130C, where the individual segmentation results 105A-105E identify distinct locations as interfaces between the blood vessel wall 115 and the blood vessel lumen 120. At the regions 130A, 130B, 130C, two or more of the individual segmentation results 105A-105E are generally divergent, and therefore, distinct locations are identified as interfaces between the blood vessel wall 115 and the blood vessel lumen 120. For example, Figure 2 As shown, at each of regions 130A, 130B, 130C, two or more (e.g., each) of the individual segmentation results 105A-105E are outside of a predetermined range of each other and are therefore considered to be generally divergent and identify significantly different locations as interfaces between the vessel wall 115 and the vessel lumen 120.

[0046] As described above, two or more of the individual segmentation results 105A-105E may be used to form the collective segmentation result 110. For example, two or more of the individual segmentation results 105A-105E may be aggregated to form the collective segmentation result 110. In the illustrated example, the median integration of the individual segmentation results 105A-105E is used to form Figure 2The collective segmentation result 110 shown. In some examples, each of the individual segmentation results 105A-105E can be used to form the collective segmentation result 110. In other examples, one or more outlier individual segmentation results 105A-105E can be discarded and the remaining individual segmentation results 105A-105E can be used to form the collective segmentation result 110. For example, in the case where an individual segmentation result differs from each of the other individual segmentation results by more than a predetermined outlier threshold, the individual segmentation result can be excluded from forming the collective segmentation result 110. In some examples, when a portion of an individual segmentation result differs from a corresponding portion of the other individual segmentation results by more than a predetermined outlier threshold, the portion of the individual segmentation result can be excluded from forming the collective segmentation result 110 (e.g., so that the corresponding portion of the collective segmentation result 110 is formed by the corresponding non-outlier portions of the other individual segmentation results).

[0047] Figure 3 An example of a cross-sectional view of a blood vessel 100 is shown, including collective segmentation results 110 presented with associated confidence levels. Figure 3 As shown, by including different indicators at regions where different error probabilities exist at the positions of the collective segmentation results 110 of the collective segmentation results 110, the collective segmentation results 110 are presented with associated confidence levels. In the example shown, visual indicators are associated with the collective segmentation results 110 to convey that there are regions where different error probabilities exist at the positions of the collective segmentation results 110 of the collective segmentation results 110. For example, the collective segmentation results 110 may include: one type of visual indicator associated with a portion of the collective segmentation results 110 where the error probability is relatively low; and another different type of visual indicator associated with a portion of the collective segmentation results 110 where the error probability is relatively high.

[0048] In the example shown, the portions of the collective segmentation result 110 include different indicators 205 , 210 , 215 .

[0049] The indicator 205 is associated with a portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position is relatively low. In the case where the error probability of the collective segmentation result 110 position is relatively low, the confidence level can be considered to be relatively high. In the illustrated example, the portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position is relatively low and is therefore associated with the indicator 205 corresponds to Figure 2 In regions 125A, 125B, 125C, the individual segmentation results 105A-105E identify similar locations as interfaces between the vessel wall 115 and the vessel lumen 120 in regions 125A, 125B, 125C.

[0050] The indicator 215 is associated with a portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position is relatively high. In the case where the error probability of the collective segmentation result 110 position is relatively high, the confidence level can be considered to be relatively low. In the example shown, the portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position is relatively high and is therefore associated with the indicator 215 corresponds to Figure 2 In regions 130A, 130C, the individual segmentation results 105A-105E identify distinct locations as interfaces between the vessel wall 115 and the vessel lumen 120 in regions 130A, 130C.

[0051] In some examples, the indicator 210 may also be included in a portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position is relatively moderate, and be associated with the portion. In the case where the error probability of the collective segmentation result 110 position is relatively moderate, the confidence level may be considered to be relatively moderate. For example, the indicator 210 may be included in a portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position transitions between relatively high and low error probabilities, and be associated with the portion. In the illustrated example, the portion of the collective segmentation result 110 where the error probability of the collective segmentation result 110 position is relatively moderate and is therefore associated with the indicator 210 may correspond to Figure 2 1 , where the individual segmentation results 105A-105E identify locations that transition between significantly different locations and significantly similar locations as interfaces between the vessel wall 115 and the vessel lumen 120.

[0052] As noted, in the illustrated example, the indicators 205, 210, 215 may be visual indicators presented on the displayed image of the cross-sectional view of the blood vessel 100. As shown, the indicators 205, 210, 215 may be located on a line representing the collective segmentation result 110. The visual indicators 205, 210, 215 may be different visual indicators, presenting different visual features, so that the user can visually distinguish between the portions of the collective segmentation result 110 associated with different confidence levels (e.g., error probabilities). For example, the different visual indicators 205, 210, 215 may each be a different color. For example, the visual indicators 205, 210, 215 may be in the form of a color gradient, such that the visual indicator 205 is displayed with a first portion of the color gradient (e.g., a color spectrum that is typically green), the visual indicator 210 is displayed with a second, subsequent portion of the color gradient (e.g., a color spectrum that is typically yellow), and the visual indicator 215 is displayed with a third, subsequent portion of the color gradient (e.g., a color spectrum that is typically red). In another example, the visual indicators 205, 210, 215 may take the form of different line patterns at the lines representing the collective segmentation result 110. For example, the different visual indicators 205, 210, 215 may each be a different dashed line pattern or a combination of different solid and dashed line patterns, so that the user can visually distinguish portions of the collective segmentation result 110 associated with different confidence levels (e.g., error probabilities). The different visual indicators 205, 210, 215 may differ in other visual characteristics.

[0053] Various techniques may be utilized to determine the confidence level associated with the collective segmentation result 110. One specific example for determining the confidence level associated with the collective segmentation result 110 is described below. Other techniques may be used to determine the confidence level associated with the collective segmentation result 110 within the scope of the present disclosure.

[0054] An exemplary technique performs multiple different segmentation algorithms to obtain corresponding individual segmentation results. Then, a clustering metric is applied to these individual segmentation results and the clustering metric is converted into a problem-specific probability metric.

[0055] In this exemplary technique, in order to obtain clustering, the ability to ensemble predictions can be exploited. Ensemble generally means aggregating the results of several models (e.g., individual segmentation results) on the same data (e.g., image data corresponding to a cross-sectional view of a blood vessel). After aggregation, a metric is used to obtain a final single result (e.g., a collective segmentation result).

[0056] In this exemplary technique, by integrating the results together, several possible boundary locations (e.g., individual segmentation results) and the final calculated boundary (e.g., collective segmentation result) can be obtained. The more results there are for a single boundary or nearby boundaries, the higher the confidence that the selected boundary is correct. If most of the results are different, it can be concluded that these boundaries are ambiguous from the image data, and therefore the confidence that the correct boundary is selected at the corresponding location is low.

[0057] In the integrated process of the exemplary technique, the image segmentation confidence level can be determined by measuring the spread of each algorithm's results from the consistency of all algorithms (e.g., measuring the spread of each of the individual segmentation results from the set of segmentation results). Any type of measurement statistics, including the results of each algorithm (e.g., and weighted combinations thereof) and any type of definition of the consistency of all algorithm results, can be used to estimate confidence.

[0058] For example, in this exemplary technique, the standard deviation of the predictions (e.g., individual segmentation results) can be used to determine the clustering metric. A high standard deviation can be associated with a low confidence, while a low standard deviation can be associated with a high confidence. In some cases, one or more outliers, such as outliers from a single individual segmentation result, can be removed to avoid biasing the results due to a small number of incorrect predictions.

[0059] In this exemplary technique, once the standard deviation of the prediction (e.g., individual segmentation results) is determined, the standard deviation can be converted into a confidence metric. For example, the minimum and maximum standard deviations for clipping can be determined empirically for this purpose. For example, a standard deviation less than one can be considered a perfect prediction, while a standard deviation greater than ten can be considered a complete guess. After clipping to the minimum and maximum standard deviations, the determined standard deviation can be normalized to provide an error probability. The inverse of this error probability value can be considered a confidence level that the final calculated boundary (e.g., collective segmentation result) is correct.

[0060] Such a technique can be useful because it can allow the confidence of a segmentation result (e.g., a collective segmentation result) to be determined without comparing individual predictions (e.g., individual segmentation results) to ground truth (e.g., without comparing to the actual boundary between two items in the image data). Although the present disclosure describes the technique in an exemplary intravascular imaging application, the present disclosure can be applied to other imaging applications. These other imaging applications can include, for example, magnetic resonance imaging (MRI) image data, computed tomography (CT) image data, visible light image data, and thermal image data.

[0061] Figure 4 and Figure 5A schematic diagram showing an example of a method for determining a confidence level using clustering of individual segmentation results. Figure 4 The schematic diagram in shows relatively strong clustering around certain values, while Figure 5 The schematic in shows a wider spread, without strong clustering around any particular value. Figure 4 and Figure 5 In the example shown, the method attempts to predict the value at 2, and Figure 4 and Figure 5 Each point plotted in represents an independent prediction from the ensemble (e.g., a separate segmentation result).

[0062] like Figure 4 As shown, most predictions (e.g., individual segmentation results) are relatively accurate, while a single prediction shown at a value of about 5 is relatively inaccurate because it is farther from the value 2 than any other prediction. Figure 4 In the example of , there is relatively strong clustering around the value of 2. In one example, a single prediction showing a value of about 5 can be considered an outlier because it is outside the predetermined range from other predictions and may be excluded from further use. Figure 4 Visually it seems that after integration, the prediction of the final value (e.g., the collective segmentation result) has a relatively high confidence.

[0063] like Figure 5 As shown, the individual predictions (e.g., individual segmentation results) are relatively scattered, and therefore, the confidence of any single prediction is relatively low. In this example, there is no strong clustering around a particular value. Therefore, after integration, it can be determined that the confidence values ​​associated with the individual predictions shown here are relatively low.

[0064] like Figure 4 and Figure 5 In the examples shown in , evaluating the predicted clustering can have a strong correlation with the confidence level of the predictions in the absence of ground truth (e.g., in these examples, the true value is not known to be 2). The predicted clustering can be evaluated using any common clustering metric, such as the standard deviation described elsewhere in this paper.

[0065] In order to convert the clustering metric into a confidence level, a mapping between the clustering metric and the probability can be determined. In many examples, the relationship between the clustering metric and the error can be systematically estimated to suit a particular application. Using the example of the standard deviation, a suitable standard deviation can be experimentally selected to explain most of the clustering errors. After establishing this relationship, the mapping can be approximated to suit a variety of applications. For example, an example mapping can be a simple normalized metric after experimentally determining the maximum allowable standard deviation, as detailed earlier in this article. This mapping can be non-linear if appropriate for a particular application.

[0066] Figure 6 is a flow chart illustrating an example process for predicting cluster boundaries of objects within an image in accordance with one or more techniques described herein. Figure 6 The operation may be performed by a computing device (e.g. Figure 1 The following is for illustration purposes only. Figure 1 The context of computing device 2 is described Figure 6 .

[0067] According to the technology of the present disclosure, computing device 2 receives an image containing a cross-sectional view of an object (602). For each of a plurality of models, computing device 2 generates a boundary prediction of an object within the image using the corresponding model (604). Each of the plurality of models is a unique model. In addition, each boundary prediction has a plurality of boundary prediction parts. Computing device 2 generates a cluster boundary prediction of an object within the image based on the boundary prediction of each of the plurality of models (606). The cluster boundary prediction includes a plurality of cluster boundary prediction parts, each cluster boundary prediction part corresponding to one of the plurality of boundary prediction parts of each boundary prediction. For each of the plurality of cluster boundary prediction parts, computing device 2 determines a confidence level of the corresponding cluster boundary prediction part based on a comparison between the corresponding cluster boundary prediction part and the corresponding boundary prediction part of the boundary prediction (608). Computing device 2 outputs a graphical indication of each of the plurality of cluster boundary prediction parts for display on a display device (610). Each cluster boundary prediction part has a visual feature indicating a confidence level of the corresponding cluster boundary prediction part.

[0068] It should be appreciated that, according to examples, certain actions or events of any of the techniques described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described actions or events are required for practice of the technique). In addition, in some examples, actions or events may be performed simultaneously, for example, by multithreading, interrupt handling, or multiple processors, rather than sequentially.

[0069] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium or transmitted as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media, including any media that facilitates, for example, transferring a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present disclosure. A computer program product may include a computer-readable medium.

[0070] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage components, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. In addition, any connection is properly referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwaves are used to transmit instructions from a website, server or other remote source, then the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwaves are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but are directed to non-temporary, tangible storage media. Disks and optical disks used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and Blu-ray disks, where disks usually reproduce data magnetically, while disks use lasers to reproduce data optically. The above combination should also be included in the scope of computer-readable media.

[0071] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

[0072] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in a codec hardware unit, or provided by a set of interoperable hardware units including one or more processors as described above, together with appropriate software and / or firmware.

[0073] Various examples have been described with reference to certain disclosed examples. These embodiments are provided for illustration and not limitation. Those skilled in the art will appreciate that various changes, adjustments and modifications may be made without departing from the scope of the present invention.

Claims

1. A method comprising: receiving, by a processor, an image comprising a cross-sectional view of one or more objects; for each of a plurality of models, generating, by the processor and using the corresponding model, a boundary prediction for each of the one or more objects within the image, wherein each boundary prediction comprises a plurality of boundary prediction portions, and each of the plurality of models comprises a different model; generating, by the processor and based on the boundary prediction of each of the plurality of models and for each of the one or more objects, a cluster boundary prediction of a corresponding object within the image, the cluster boundary prediction comprising a plurality of cluster boundary prediction portions, each cluster boundary prediction portion corresponding to one of the plurality of boundary prediction portions of each of the boundary predictions; For each of the plurality of cluster boundary prediction portions for each of the one or more objects, determining, by the processor and based on a comparison between the corresponding cluster boundary prediction portion and a corresponding boundary prediction portion, a confidence level of the corresponding cluster boundary prediction portion of the corresponding object, wherein determining the confidence level of the corresponding cluster boundary prediction portion of the corresponding object comprises: for each of the plurality of models, determining, by the processor and based on a comparison between the corresponding cluster boundary prediction portion and a corresponding boundary prediction portion of the boundary prediction of the corresponding object, a dispersion of each of the corresponding boundary prediction portions of the boundary prediction of the corresponding object; and determining, by the processor and based on the dispersion of each of the corresponding boundary prediction portions of the boundary prediction of the corresponding object, a confidence level of the corresponding cluster boundary prediction portion; and For each of the one or more objects, a graphical indication of each of the plurality of cluster boundary prediction portions of the corresponding object is output by the processor for display on a display device, wherein each cluster boundary prediction portion has a visual feature indicating a confidence level of the corresponding cluster boundary prediction portion.

2. The method according to claim 1, wherein the multiple models include one or more of a machine learning model, a deep learning model, a pattern recognition model, a computer vision model, and a parameter correction model.

3. The method of claim 1 , wherein determining a spread of each of the corresponding boundary prediction portions of the boundary prediction comprises: creating, by the processor and using each of the cluster boundary prediction portions and corresponding boundary prediction portions of the boundary predictions of the respective objects, a prediction distribution, wherein the cluster boundary prediction portions are used as an average of the prediction distribution; determining, by the processor, a standard deviation for each of corresponding boundary prediction portions of the boundary prediction of the corresponding object; as well as A z-score is determined, by the processor, for each of the corresponding boundary prediction portions of the boundary prediction for the corresponding object as the number of standard deviations that the respective corresponding boundary prediction portion is from the cluster boundary prediction for the corresponding object.

4. The method according to claim 1, further comprising: determining, by the processor, that a spread of a first corresponding boundary prediction portion of a first boundary prediction of the corresponding object exceeds a deviation threshold of a prediction distribution, the first boundary prediction being associated with a first model of the plurality of models, wherein the deviation threshold comprises one of a user-defined threshold or a default threshold; removing, by the processor, a first model from the plurality of models to generate a subset of models of the corresponding object; as well as At least the corresponding cluster boundary prediction portion of the corresponding object is recomputed by the processor using only the model subset. 5 . The method of claim 1 , wherein the visual features include one or more of line color, fill pattern, line thickness, line segmentation, text indication, or graphic icon.

6. The method according to claim 1, further comprising: determining, by the processor, for each of the plurality of models, a weight of the corresponding model; as well as For each of the one or more objects, the processor generates the cluster boundary prediction of the corresponding object based on the corresponding boundary prediction portion of the boundary prediction and the weight of each model of the multiple models as a weighted cluster boundary prediction of the corresponding object within the image. The method of claim 1 , wherein the plurality of models comprises three or more different models.

8. A computing device comprising: at least one processor; as well as a storage component configured to store instructions executable by at least one processor to: receiving an image including a cross-sectional view of one or more objects; for each of a plurality of models, generating a boundary prediction for each of the one or more objects within the image using the corresponding model, wherein each boundary prediction comprises a plurality of boundary prediction portions and each of the plurality of models comprises a different model; generating, based on the boundary prediction of each of the plurality of models and for each of the one or more objects, a cluster boundary prediction of the corresponding object within the image, the cluster boundary prediction comprising a plurality of cluster boundary prediction portions, each cluster boundary prediction portion corresponding to one of the plurality of boundary prediction portions of each of the boundary predictions; For each of the one or more objects, for each of the plurality of cluster boundary prediction parts of the corresponding object, determine a confidence level of the corresponding cluster boundary prediction part of the corresponding object based on a comparison between the corresponding cluster boundary prediction part and a corresponding boundary prediction part, wherein determining the confidence level of the corresponding cluster boundary prediction part of the corresponding object comprises: for each of the plurality of models, determining a dispersion of each of the corresponding boundary prediction parts of the boundary prediction of the corresponding object based on a comparison between the corresponding cluster boundary prediction part and a corresponding boundary prediction part of the boundary prediction of the corresponding object; and determining a confidence level of the corresponding cluster boundary prediction part based on the dispersion of each of the corresponding boundary prediction parts of the boundary prediction of the corresponding object; and For each of the one or more objects, a graphical indication of each of the plurality of cluster boundary prediction portions of the corresponding object is output for display on a display device, wherein each cluster boundary prediction portion has a visual feature indicating a confidence level of the corresponding cluster boundary prediction portion.

9. The computing device of claim 8, wherein the plurality of models comprises one or more of a machine learning model, a deep learning model, a pattern recognition model, a computer vision model, and a parameter correction model.

10. The computing device of claim 8, wherein the instructions executable by one or more processors to determine a spread of each of the corresponding boundary prediction portions of the boundary predictions include instructions executable by the one or more processors to: creating a prediction distribution using each of the cluster boundary prediction portions and corresponding boundary prediction portions of the boundary prediction of the corresponding object, wherein the cluster boundary prediction portion serves as an average of the prediction distribution; determining a standard deviation for each of corresponding boundary prediction portions of the boundary prediction of the corresponding object; and A z-score is determined for each of the corresponding boundary prediction portions of the boundary prediction of the corresponding object as the number of standard deviations that the corresponding boundary prediction portion is from the cluster boundary prediction portion of the corresponding object.

11. The computing device of claim 8, wherein the instructions are further executable by one or more processors to: determining that a spread of a first corresponding boundary prediction portion of a first boundary prediction of the corresponding object exceeds a deviation threshold of a prediction distribution, the first boundary prediction being associated with a first model of the plurality of models; removing the first model from the plurality of models to generate a subset of models of the corresponding object; as well as At least a portion of the corresponding cluster boundary predictions for the respective objects is recomputed using only the model subset. 12 . The computing device of claim 11 , wherein the deviation threshold comprises one of a user-defined threshold or a default threshold.

13. The computing device of claim 8, wherein the visual feature comprises one or more of a line color, a fill pattern, a line thickness, a line segmentation, a textual indication, or a graphical icon.

14. The computing device of claim 8, wherein the instructions are further executable by one or more processors to: For each of the plurality of models, determining a weight of the corresponding model; and For each of the one or more objects, the cluster boundary prediction of the corresponding object is generated as a weighted cluster boundary prediction of the corresponding object within the image based on a corresponding boundary prediction portion of the boundary prediction and a weight of each of the plurality of models.

15. The computing device of claim 8, wherein the plurality of models comprises three or more different models.

16. The computing device of claim 8, further comprising a camera configured to capture the image.

17. A non-transitory computer-readable storage medium is described as having stored thereon instructions which, when executed, cause one or more processors of a computing device to: receiving an image including a cross-sectional view of one or more objects; for each of a plurality of models, generating a boundary prediction for each of the one or more objects within the image using the corresponding model, wherein each boundary prediction comprises a plurality of boundary prediction portions and each of the plurality of models comprises a different model; generating, based on the boundary prediction of each of the plurality of models and for each of the one or more objects, a cluster boundary prediction for the corresponding object within the image, the cluster boundary prediction comprising a plurality of cluster boundary prediction portions, each cluster boundary prediction portion corresponding to one of the plurality of boundary prediction portions of each of the boundary predictions; For each of the plurality of cluster boundary prediction parts for each of the one or more objects, determine a confidence level of the corresponding cluster boundary prediction part of the corresponding object based on a comparison between the corresponding cluster boundary prediction part and a corresponding boundary prediction part, wherein determining the confidence level of the corresponding cluster boundary prediction part of the corresponding object comprises: for each of the plurality of models, determining a dispersion of each of the corresponding boundary prediction parts of the boundary prediction of the corresponding object based on a comparison between the corresponding cluster boundary prediction part and a corresponding boundary prediction part of the boundary prediction of the corresponding object; and determining a confidence level of the corresponding cluster boundary prediction part based on the dispersion of each of the corresponding boundary prediction parts of the boundary prediction of the corresponding object; and For each of the one or more objects, a graphical indication of each of the plurality of cluster boundary prediction portions of the corresponding object is output for display on a display device, wherein each cluster boundary prediction portion has a visual feature indicating a confidence level of the corresponding cluster boundary prediction portion.

Images