Method of refining segmentation mask

Abstract

To provide a computer-implemented method of refining a segmentation mask.SOLUTION: The method includes: providing a first segmentation mask associated with an image; providing a user input comprising a user-defined area of the image; and obtaining a second segmentation mask by refining the first segmentation mask based on image data of the image using a computer-implemented algorithm, where the algorithm is configured to penalize a violation of the user input.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Image segmentation encompasses a variety of functions for processing image data, such as image detection, image content detection, or image classification.

[0002] In medical image processing methods, image segmentation can be useful, for example, for automated or semi-automated lesion or ROI (region of interest) detection. Examples of medical image processing methods include ultrasound imaging, mammography, tomosynthesis, etc.

[0003] Computational image segmentation has become important in fields such as cardiology, oncology, and radiology. With the advent of deep learning and other artificial intelligence (AI) techniques, the ability of computers to perform vision tasks such as identification, localization, and generation has become widespread. For example, convolutional neural networks (CNNs) have proven particularly effective for the task of image segmentation and can therefore be leveraged to solve lesion segmentation tasks in medical files, as well as other fields.

[0004] However, such techniques may not be perfect, so that examiners, e.g., physicians, physicists, radiologists, etc., may wish or have to review and possibly modify the results in the light of their own skills and experience.

[0005] Conventional solutions for manual post-processing of automatically predicted segmentation masks are rather tedious and unintuitive, which ultimately results in time-consuming and effort-intensive.

[0006] In view of the above, there exists a need to address these and related shortcomings. Summary of the Invention

[0007] A computer-implemented method for refining a segmentation mask is provided, the method including the steps of providing a first segmentation mask associated with an image, providing a user input including a user-defined region of the image, and refining the first segmentation mask based on image data of the image with a computer-implemented algorithm to obtain a second segmentation mask, the algorithm being configured to penalize violations of the user input.

[0008] Providing such a method allows improving the first segmentation mask based on user hints, i.e. user input. For example, a user can input hints regarding regions to include or exclude region(s) from the first segmentation mask in a fuzzy way. The acquisition of the second segmentation mask may be automatically guided by the user hints. According to the method, the first segmentation mask may be user-adaptive. Thereby, the adaptation or refinement of the first segmentation mask can be considered to be at least semi-automatic, saving the user time and effort, since the user only needs to provide fuzzy user hints instead of exact correction of the mask.

[0009] Therefore, the user inputs are sometimes called user constraints.

[0010] The first segmentation mask can be generated automatically by a computer-implemented algorithm, for example an AI-based algorithm.

[0011] The first segmentation mask may be associated with a region of interest in the image. The region of interest may be associated with only a portion of the image. Thus, the first segmentation mask may for example cover only a portion of the image, i.e. a region of interest (ROI).

[0012] The user-defined region of the image may be adapted by the user to modify the first segmentation mask, in other words, the user-defined region of the image may be determined and / or adjusted by the user to modify the first segmentation mask.

[0013] Modifying the segmentation mask means expanding the first segmentation mask and / or shrinking the first segmentation mask.

[0014] For example, the user-defined region may include a first user-defined region adapted by a user to extend the first segmentation mask.

[0015] In a further example, the user-defined region may include a second user-defined region adapted by a user to restrict the first segmentation mask.

[0016] The algorithm may be configured to modify the first segmentation mask in a fuzzy manner based on the user-defined regions by applying a penalty rule. The penalty rule may be predefined or may be defined or selected by a user or a system while the method of the present disclosure is performed. Also, the penalty rule may be preset and / or predefined and adapted or adaptable while the method of the present disclosure is performed.

[0017] For example, the user-defined region is used as a soft reference against which the first segmentation mask is modified.

[0018] For example, the algorithm may be configured to include the first user-defined region in the first segmentation mask in a fuzzy manner by applying a penalty rule.

[0019] In a further example, the algorithm may be configured to exclude the second user-defined region from the first segmentation mask in a fuzzy manner by applying a penalty rule.

[0020] The algorithm includes at least a penalty rule.

[0021] According to one example, penalty rules may be configured to penalize violations of user-defined domains.

[0022] According to a further embodiment, the penalty rule may be configured to penalize a violation of extending the first segmentation mask by the first user-defined region. In other words, this case concerns a condition where the first segmentation mask, which is improved based on the image data, does not match the first user-defined region. Therefore, the improvement of the basic segmentation mask based on the image data may be considered as a violation of the user input. This violation may be penalized.

[0023] According to a further embodiment, the penalty rule can be configured to penalize a violation of excluding the second user-defined region from the first segmentation mask. In other words, this case concerns a condition where the first segmentation mask, which is improved based on the image data, does not completely exclude the second user-defined region. Therefore, the improvement of the basic segmentation mask based on the image data may be considered as a violation of the user input. This violation may be penalized.

[0024] The user input may further include a user-defined confidence level that may be associated with the user-defined region. The algorithm may be configured to penalize violations of the user input as a function of the user-defined confidence level.

[0025] For example, parameters can be provided in the method to allow the confidence in the user input to be adjusted, for example manually by a user. In other words, the user input can be accommodated in a fuzzy framework to allow for adjustable confidence in the automatic segmentation, as described below. This advantageously allows for a more flexible tolerance for user error.

[0026] The image data may include statistical measures related to image features, such as low-level image features, which may include, for example, at least one of contours, edges, angles, intensity, color, motion, and / or texture.

[0027] The image data may be used by an algorithm to discriminate between image regions to refine the first segmentation mask.

[0028] The algorithm may be configured to utilize local likelihood statistics and / or context distributions based on the image data to adjust the segmentation boundary of the first segmentation mask.

[0029] An algorithm is a predefined function

number

[0030] The function F(Φ) may be configured to penalize violations of the user input. Thus, the function F(Φ) may represent, for example, a fuzzy penalty for violations of the user input by the segmentation.

[0031] The function R(Φ) may represent a regularization penalty term for the segmentation mask.

[0032] An algorithm, e.g., a function D(Φ), may be configured to compare a local distribution around each of a plurality of image points with a distribution in an image region inside a first segmentation mask and with a distribution in an image region outside the first segmentation mask, in which case points of similar contextual information may be grouped together in a second segmentation mask.

[0033] For example, the algorithm can use an optimal transport metric (e.g., the Weisserstein metric) to compare the distributions. For example, the optimal transport metric can be defined by a function D(Φ):

[0034] For example, the optimal transport metric is a predefined function

number

number

[0035] The image points may include image pixels, image voxels, or image points of any dimension.

[0036] The algorithm, e.g. the function D(Φ), may be configured to take into account local likelihood information, such as the likelihood of points from image regions inside the first segmentation mask competing with the likelihood of points from image regions outside the first segmentation mask, thereby separating points from inner and outer image regions.

[0037] For example, the function D(Φ) has the terms:

number

[0038] The penalty rule can be, for example, a predefined function

number

[0039] An algorithm, for example a function R(Φ), can be configured to smooth the segmentation boundary of the second segmentation mask. The function R(Φ) can be, for example, a predefined function

number

number

[0040] So, according to your particular example, the predefined function

number

number

[0041] The first segmentation mask may be automatically generated by a computer-implemented algorithm.

[0042] The images are medical images and / or images obtained by a medical imaging system, optionally an ultrasound imaging system.

[0043] According to one or more examples, the first and / or second segmentation mask may be displayed on a display device.

[0044] The first segmentation mask may be displayed on a display device.

[0045] User input may be received via an input device.

[0046] The user input and the first segmentation mask may be provided to a computing device.

[0047] The computer-implemented algorithm may be executed by the computing device to obtain a second segmentation mask.

[0048] The second segmentation mask is displayed on the display device, optionally adjacent to or in place of the first segmentation mask.

[0049] The method can be iterated, for example the second segmentation mask of iteration N can be used as the first segmentation mask of iteration N+1.

[0050] The method, and for example the computer-implemented algorithm, can be performed at least partially in real-time or pseudo-real-time, for example obtaining the second segmentation mask by improving the first segmentation mask, such that a user providing a user input does not perceive a significant delay between entering the user input and obtaining the second segmentation mask.

[0051] The disclosure may further relate to a method of generating a training data set for an artificial intelligence (AI) algorithm, the method including: - applying the method according to any one of the preceding claims to a plurality of first segmentation masks to obtain a plurality of second segmentation masks; Generating a training data set based on the plurality of second segmentation masks and, optionally, the plurality of first segmentation masks.

[0052] The disclosure may further relate to a method of training an artificial intelligence (AI) algorithm, the method including: performing the method of the preceding claim to obtain a training data set; Training an artificial intelligence (AI) algorithm in a supervised manner using the training data, where a plurality of first segmentation masks are used as inputs during training and a plurality of second segmentation masks are used as target outputs.

[0053] Examples of artificial intelligence (AI) algorithms may include machine learning models such as decision trees, random forests, logistic regression, gradient boosting, neural networks such as convolutional neural networks (CNNs), and / or support vector machines. Predefined algorithms may be used to perform classification and prediction tasks such as image recognition. For example, a decision tree algorithm may be used to classify images of objects based on their features. Other examples of machine learning models may consist of deep learning models such as neural networks, k-nearest neighbors, and / or clustering algorithms, which may be used for tasks such as anomaly detection and data segmentation. AI may refer to the development of computer systems and algorithms that perform tasks that typically require human intelligence, such as visual recognition. AI systems and algorithms may be trained with large datasets and sophisticated algorithms that allow them to identify patterns and trends, make predictions, and optimize outcomes.

[0054] The present disclosure may further refer to a computing device comprising: At least one processor; At least one memory storing computer-executable instructions that, when executed by the processor, cause the computing device to perform a method according to any one of the preceding claims.

[0055] For example, the computing device may be configured to calculate, or receive from an external system, a first segmentation mask associated with the image; receiving user input including a user-defined region of the image; A second segmentation mask is obtained or calculated by refining the first segmentation mask based on image data of the image using a computer-implemented algorithm, where the algorithm is configured to penalize violations of user input.

[0056] The computer-executable instructions may constitute, for example, a computer-implemented algorithm.

[0057] The computing device may be configured to be associated with a display device, and the first and / or second segmentation masks may be displayed on the display device.

[0058] The computing device may be configured to be associated with an input device configured to receive and provide user input to the computing device.

[0059] A processor (or processing unit) may be a component of an electronic device that may be responsible for performing computational tasks. A processing device may be or include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and / or an Application-Specific Integrated Circuit (ASIC). Methods according to the present disclosure may also be performed on a virtual server.

[0060] The present disclosure may also relate to a system for generating a segmentation mask, the system comprising means for performing a method according to any embodiment of the present disclosure, for example the system may include or be a computing device as described above.

[0061] The present disclosure may also relate to an imaging system (eg, ultrasound imaging) including means for performing the method according to any embodiment of the present disclosure.

[0062] The present disclosure may also relate to a computer program comprising instructions that, when said program is executed by a computer, cause the computer to carry out the method according to any embodiment of the present disclosure.

[0063] The present disclosure may also relate to a computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform a method according to any embodiment of the present disclosure.

[0064] Except where inconsistent, it is contemplated that combinations of the above elements with those herein are possible.

[0065] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are provided for purposes of explanation and are not restrictive of the disclosure, as claimed.

[0066] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to support and explain the principles thereof. [Brief description of the drawings]

[0067] [Figure 1] FIG. 1 illustrates a schematic diagram of a method for improving a segmentation mask according to an embodiment of the present disclosure.

[0068] [Diagram 2] FIG. 2 illustrates a method for obtaining a second segmentation mask according to an embodiment of the present disclosure.

[0069] [Diagram 3] FIG. 3 illustrates a method for training an artificial intelligence (AI) algorithm according to an example of the present disclosure.

[0070] [Figure 4] FIG. 4 illustrates a method for providing and processing user input according to an example of this disclosure.

[0071] [Figure 5A] FIG. 5A shows a schematic example of an image according to operation S1 of the method of FIG.

[0072] [Figure 5B] FIG. 5B shows a schematic example of a first segmentation mask according to operations S2 and S3 of the method of FIG.

[0073] [Figure 5C] FIG. 5C shows a schematic example of a first segmentation mask and user input according to operation S4 of the method of FIG.

[0074] [Figure 5D] FIG. 5D shows a schematic example of a second segmentation mask according to operations S5 and S6 of the method of FIG.

[0075] [Figure 6] FIG. 6 is a schematic diagram of an ultrasound system 10 according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0076] Description of the drawings Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0077] 1 shows a schematic diagram of a method for improving a segmentation mask according to an embodiment of the present disclosure. The method may include a number of operations. One or more operations may be implemented, for example, by respective software modules.

[0078] In the first optional operation S1, an image can be provided. FIG. 5A shows an example of an image 300a. The image can be provided, for example, by an imaging system, for example a medical imaging system. The image can be associated with a medium scanned by the imaging system. For example, the imaging system can comprise at least one of mammography, tomosynthesis, magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT) scan, positron emission tomography (PET) scan, optical coherence tomography (OCCT), optical coherence tomography (OCT), x-ray examination, and ultrasound imaging. An exemplary ultrasound system is shown in FIG. 6. In general, the image is acquired by a first system performing the method of FIG. 1, or prepared or acquired by a second system and provided to the first system.

[0079] In operation S2, a first segmentation mask is provided. For example, in operation S2, the first segmentation mask may be obtained based on image data of the image of operation S1. The first segmentation mask may be associated with a region of interest (ROI) in the image. The ROI may be or may include a lesion in a medium, for example human or animal tissue.

[0080] In one example, the first segmentation mask can be obtained by detecting and segmenting a region of interest (ROI) in the medium. A predefined computer-implemented segmentation algorithm can be used, for example an AI-based algorithm, such as a neural network, such as a convolutional neural network (CNN). The region of interest may be associated with only a portion of the image. Thus, the first segmentation mask can cover, for example, only a portion of the image, i.e., the region of interest (ROI).

[0081] However, it is also possible that the image and / or the first segmentation mask are simply obtained from a communication interface, a data storage device, etc. For example, the image and / or the first segmentation mask may be predetermined and / or pre-stored and provided to a system performing a method according to the present disclosure.

[0082] In an optional operation S3, the first segmentation mask can be displayed, for example, on top of the image, as in the example shown in FIG.

[0083] In operation S4, user input is provided. The user input may include a user-defined region of the image, for example as shown in Fig. 5C. The user-defined region of the image may be provided by a user to modify the first segmentation mask.

[0084] In operation S5, a second segmentation mask is obtained by using a computer implemented algorithm to refine the first segmentation mask based on image data of the image, whereby the algorithm penalizes violations of the user input.

[0085] In an optional operation S6, a second segmentation mask may be displayed, optionally adjacent to or in place of the first segmentation mask on the display device, the latter case being shown in the example of FIG.

[0086] The method may be iterated in operation S7. For example, the second segmentation mask of iteration N is used as the first segmentation mask of iteration N+1. Thus, the user can continuously improve the segmentation mask by modifying the (first) segmentation mask in each iteration.

[0087] The method may be performed at least partially in real time, for example obtaining the second segmentation mask in operation S5 may be performed in real time or near real time such that a user providing user input does not perceive a significant delay in concentrating on the image displayed on the display device.

[0088] Fig. 2 illustrates a method for obtaining a second segmentation mask according to an embodiment of the present disclosure. The method of Fig. 2 may correspond to operation S5 of the method of Fig. 1 or may be a specific example of operation S5. The method of Fig. 2, and in particular operation S5, may be performed by a computer-implemented algorithm according to the present disclosure.

[0089] The operation S5 may include a basic refinement operation S5a1, in which the first segmentation mask is basically refined, for example based on the image data.

[0090] The image data may include statistical criteria for image features, such as low-level image features. The image features may include, for example, at least one of contours, edges, angles, intensity, color, motion, and / or texture. The image data may be used by an algorithm to discriminate between image regions to improve the first segmentation mask.

[0091] Operation S5a1 may be performed using an algorithm as described below or some rules defined by an algorithm. The algorithm may be configured to utilize local likelihood statistics and / or context distributions based on the image data to adjust the segmentation boundary of the first segmentation mask. The algorithm may be adaptively configured based on characteristics of the medium scanned by the image processing system, for example the type of medium (e.g. type of organ such as liver or breast).

[0092] An algorithm is a predefined function.

number

[0093] The function D(Φ) may represent a penalty term configured to penalize a decreased fit of the first segmentation mask compared to the image data. Thus, the function D(Φ) may represent a penalty for a poor fit of the first segmentation mask compared to the image data. Thus, the penalty by the function D(Φ) may be increased when the fit is decreased (i.e., when the fitness is relatively decreased).

[0094] The function F(Φ) may be configured to penalize violations of the user input. Thus, the function F(Φ) may represent, for example, a fuzzy penalty for violations of the user input by the segmentation.

[0095] The function R(Φ) represents the regularization penalty term for the segmentation mask.

[0096] An algorithm, e.g., a function D(Φ), may be configured to compare a local distribution around each of a plurality of image points with a distribution in an image region inside a first segmentation mask and a distribution in an image region outside the first segmentation mask, in which case points of similar contextual information may be grouped together in a second segmentation mask.

[0097] For example, the algorithm can use an optimal transport metric (e.g., the Weisserstein metric) to compare the distributions. For example, the optimal transport metric can be defined by a function D(Φ):

[0098] In one example, the optimal transport metric is a predefined function

number

number

[0099] An image point may include an image pixel, an image voxel, or an image point of any dimension.

[0100] The algorithm, e.g., function D(Φ), may be further configured to take into account local likelihood information, such as the likelihood of points from image regions inside the first segmentation mask competing with the likelihood of points from image regions outside the first segmentation mask, thereby separating points in regions inside the image from regions outside the image.

[0101] For example, the function D(Φ) may include the following terms:

number

[0102] An algorithm, for example a function R(Φ), can be configured to smooth the segmentation boundary of the second segmentation mask. The function R(Φ) can be, for example, a predefined function

number

number

[0103] The operations S5 and / or S5a1 include an operation S5a2 for penalizing violations of the user input with the basic improvement operations.

[0104] For example, the algorithm may include penalty rules that may be configured to penalize violations of the user-defined domain.

[0105] The user-defined region may be used, for example, as a soft criterion by which the first segmentation mask may be modified. For example, the algorithm may be configured to include the first user-defined region in the first segmentation mask in a fuzzy manner by applying a predefined penalty rule. In a further example, the algorithm is configured to exclude the second user-defined region from the first segmentation mask in a fuzzy manner by applying a penalization rule.

[0106] The penalty rule can be, for example, a predefined function

number

number

[0107] Thus, the computer-implemented algorithm may be configured to essentially refine the first segmentation mask based on the image data of the image, as in operation S5a1. However, in this context, the algorithm penalizes violations of the user input, i.e., constrains the basic refinement based on the image data to take the user input into account (see S5a2). Thus, if the user-defined region does not match the basic refinement, the basic refinement is constrained to better respect the user-defined region.

[0108] Operations S5, S5a1 and / or S5a2 may include an optional operation S5a3 that takes into account a user-defined trust level.

[0109] For example, the user input may further include a user-defined trust level to be associated with the user-defined region.

[0110] The algorithm may be configured to penalize violations of user input as a function of a user-defined confidence level.

[0111] For example, parameters may be provided to the method such that the confidence in the user input may be adjusted manually, for example by a user.

[0112] The penalty rule is, in the particular example of (5), e.g., the predefined function

number

number

[0113] FIG. 3 illustrates an exemplary method for training an artificial intelligence (AI) algorithm according to an example of the present disclosure.

[0114] In operation S11, a set (i.e., a plurality) of first segmentation masks may be provided. Optionally, the operation may include providing images associated with the first segmentation masks, respectively. The images are preferably different from each other and / or the segmentation masks are preferably different from each other.

[0115] The first segmentation mask may correspond to the segmentation mask of operation S2 of Figure 1. Thus, operation S11 may correspond to multiple operations S2 providing different first segmentation masks.

[0116] In operation S12, a plurality of second segmentation masks may be obtained. The second segmentation masks may correspond to the segmentation masks of operation S5 of Fig. 1. Thus, operation S12 may correspond to a plurality of operations S5 providing different second segmentation masks. In other words, the second segmentation masks may be obtained using the method according to the present disclosure.

[0117] In operation S13, a training data set may be generated. The training data set may include a set of samples. Each sample may include a first segmentation mask and / or an associated second segmentation mask, optionally with an associated image.

[0118] In operation S14, an artificial intelligence (AI) algorithm may be trained using the training data set. Examples of artificial intelligence (AI) algorithms may include decision trees, random forests, logistic regression, gradient boosting, neural networks such as convolutional neural networks (CNNs), and / or machine learning models such as support vector machines.

[0119] For example, an AI algorithm may be trained in a supervised manner using the first segmentation mask and / or the respective image as input and the respective second segmentation mask as a target output.

[0120] In this manner, the AI ​​algorithm can be trained to predict a second, improved segmentation mask based on the image and / or the first segmentation mask. Thus, the trained AI algorithm can automate, or at least partially automate, the manual task of providing user input.

[0121] FIG. 4 illustrates a method for providing and processing user input according to an example of this disclosure.

[0122] The method of Figure 4 may correspond to, or be a specific example of, operation S4 of the method of Figure 1. The method of Figure 2, and in particular operation S4, may be performed by a software module configured to receive and optionally process user input.

[0123] Operation S4 comprises operation S4a of providing a user-defined region of the image. For example, the user-defined region may be provided by a user manually drawing the region using a user interface device associated with the computing device on which the method is being performed. The user interface device may be, for example, a touch screen device (e.g., the control panel display screen 49 illustrated in FIG. 6).

[0124] In operation S4b, it may be determined whether the user-defined region is intended to extend or restrict the first segmentation mask. For example, the method or the respective software module may be associated with additional input means that may be used by the user to select the type of the user-defined region. The additional input means may for example be one or more physically or software-implemented buttons of a user interface device.

[0125] The type of the user-defined region can also be automatically detected by the method. For example, if 50% or more of the user-defined region is outside the first segmentation mask (e.g., see user-defined region 304a shown in FIG. 5C), it may be determined that the user-defined region is intended to extend the first segmentation mask. Furthermore, if 50% or more of the user-defined region is inside the first segmentation mask (e.g., see user-defined region 304b shown in FIG. 5C), it is determined that the user-defined region is intended to be removed from the first segmentation mask.

[0126] Thus, in operation S4b(o.1), i.e. option 1 of operation S4b, the user-defined region may be labeled as a first user-defined region intended to be added to the first segmentation mask. Further, in operation S4b(o.2), i.e. option 2 of operation S4b, the user-defined region may be labeled as a second user-defined region intended to be removed from the first segmentation mask.

[0127] Further, operation S4 may include providing a user-defined trust level for each provided user-defined region. For example, a user-defined trust level may be associated with the user-defined region. In one example, the user-defined trust level may be a number between 0 and 100 that may be manually entered by a user using a user interface device.

[0128] The user-defined regions, in particular the first and / or second user-defined regions, optionally together with one or several confidence levels, may then be passed to operation S5 or each of the above-mentioned algorithms, in which operation S5 a second segmentation mask is obtained.

[0129] The methods of Figures 1, 2 and 4 are described in more detail below in conjunction with the examples of Figures 5A-5D.

[0130] FIG. 5A shows a schematic example of an image according to operation S1 of the method of FIG.

[0131] The image 300 may be, for example, an ultrasound image, as described above in the context of operation S1 of Fig. 1, or any other type of image. In particular, the image may represent an ultrasound scanned medium. The medium may be, for example, a human body, an animal or a part of matter. The image may be displayed on a display device, as described above and in the context of Fig. 6 below.

[0132] The image may include a region of interest 302, e.g., a region suspected of being a lesion in the medium. In one embodiment, the region of interest may be manually selected by a user by drawing a bounding box 301 that constitutes the region of interest. In other embodiments, the region of interest may be automatically determined or detected.

[0133] FIG. 5B shows a schematic example of a first segmentation mask according to operations S2 and S3 of the method of FIG.

[0134] The first segmentation mask 303 may be determined automatically, as described above in the context of operation S2 of Fig. 1, or it may be determined based on a manually drawn bounding box 301.

[0135] The first segmentation mask 303 may be displayed together with the image 300, in particular by overlaying it on the image 300. The first segmentation mask 303 may thus indicate the boundaries of a region of interest of a possible lesion in the scanned medium, muscle, organ or part thereof.

[0136] A user, e.g. an examiner such as a physician or radiologist, may wish to review and possibly adapt the first segmentation mask 303 taking into account his or her skills and experience. In particular, the user may wish to modify the boundaries of the first segmentation mask by local expansion and / or limitation of the mask. An example of addressing this issue is shown in Figures 5C and 5D.

[0137] FIG. 5C shows a schematic example of a first segmentation mask and user input according to operation S4 of the method of FIG.

[0138] The systems and methods can provide means to allow a user to interact to provide user-defined regions likely to be included in the ROI and regions likely to be excluded.

[0139] In this example, the user has provided two user-defined regions 304a and 304b, for example by drawing a line on the display device and / or by setting one or more marker points 304c that define the boundaries of the user-defined regions.

[0140] 4, based on manual user selection and / or based on an automatic determination, user-defined region 304a may be labeled as a first user-defined region intended to be added to the first segmentation mask, and user-defined region 304b may be labeled as a second user-defined region intended to be removed from the first segmentation mask.

[0141] Furthermore, as explained in the context of FIG. 4, the user may additionally input a user-defined trust level (or levels) for one or each or some of the user-defined regions 304a and 304b.

[0142] FIG. 5D shows a schematic example of a second segmentation mask according to operations S5 and S6 of the method of FIG.

[0143] A second refined segmentation mask 305 is derived based on the image data of image 300 using a computer-implemented algorithm, the algorithm configured to penalize violations of the user input.

[0144] Thus, an image region 305a is included in the second segmentation mask compared to the first segmentation mask 303. The added image region 305a is determined by an algorithm based on the image data, where a user input (i.e., the first user-defined region 304a) can be used as a fuzzy user input.

[0145] Correspondingly, image region 305b has been removed from the second segmentation mask compared to the first segmentation mask 303. The added image region 305a is determined by an algorithm based on the image data, where the user input (i.e., the second user-defined region 304b) is used as a fuzzy user input.

[0146] It should be noted that the user input does not need to be very accurate: the statistical terms of the algorithm according to the present disclosure can provide more accurate segmentation boundaries, while the user constraints can be taken into account in a fuzzy way.

[0147] FIG. 6 is a schematic diagram of an ultrasound system 10 according to an embodiment of the present disclosure.

[0148] The ultrasound system 10 may be, for example, a system for improving a segmentation mask according to the present disclosure. The system may be configured, for example, to perform the method according to the present disclosure. However, the method according to the present disclosure may also be performed (at least in part) by an external system.

[0149] The ultrasound imaging system 10 may include: Probe 20, a processing unit 30 for processing an image based on signals received by the probe; a control panel 40a connected to the processing unit, the control panel including at least a button 41 and a touch pad 42; and A display 50 for visualizing the image.

[0150] The probe 20 may be associated with a processing unit 30 via a cable 21 and / or via a wireless connection(s) and may emit ultrasound waves W into a medium M and receive ultrasound waves W from the medium M, the received ultrasound waves being the result of or resulting from reflection of the emitted ultrasound waves on scattering particles within the medium M.

[0151] The display screen 50 may be a screen for visualizing the image processed by the processing unit 30. The display 50 may also visualize other information such as the scale used in the image and / or configuration information for processing or any information such as help information for the touchpad 42 or contextual gesture help.

[0152] The display screen can be articulated on the support arm 51 for better positioning for the user. The display screen is typically a large size (at least 20 inches) high definition screen for better image visualization for the user.

[0153] The control panel 40a is, for example, a portion of the system casing 31, which includes a panel casing having a substantially flat surface inclined toward the user for one-handed operation by the user. The control panel 40a can be moved up and down by a hanger to accommodate the size of the user, and can be moved forward and backward optionally to accommodate the position of the user. The control panel 40a can include a control panel display screen 49 for visualizing some setting information or any information dedicated to the user.

[0154] The processing unit 30 may be configured to improve the segmentation mask according to the present disclosure. The processing unit may also transmit data to an external device in order to at least partially execute the method of the present disclosure on the external device. Examples of external devices include a server, a computer, a dedicated workstation, a device for displaying images obtained from an electronic control device, or any other external device. Thus, the method according to the present disclosure may be performed by at least one of the processing device and the external device. Furthermore, the step for constructing image data based on the acquired data, i.e. the step for compiling ultrasound image data, may be performed by the same processing device as the processing device for optimizing the optimization step, or may be performed (at least in part) by a different processing device.

[0155] According to further examples, the system 10 may include at least one processing unit (or processor) and memory. In examples, the processor and memory unit may be a computer or computers that may be incorporated into the system or communicatively linked thereto. Depending on the exact configuration and type of computing device, the memory (storing instructions for evaluating ultrasound data or performing the methods described herein) may be volatile (such as RAM), non-volatile (such as RAM, flash memory, etc.), or a combination thereof. Additionally, the system 10 may also include storage devices (removable and / or non-removable) including, but not limited to, magnetic or optical disks or tapes. Similarly, the system 10 may have input device(s) such as a keyboard, mouse, pen, voice input, and / or output device(s) such as a display, speaker, printer, etc. The environment may also include one or more communication connections such as 4G / 5G, WIFI, LAN, WAN, point-to-point, etc. In embodiments, the connections may be operable to provide point-to-point communication, connection-oriented communication, connectionless communication, etc.

[0156] The system 10 may typically include some form of computer-readable media. Computer-readable media may be any available media accessible by a processing unit (or processor) or other devices constituting the operating environment. By way of example, and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media does not include communication media.

[0157] Communication media embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, microwave, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable media.

[0158] System 10 may be a single computer operating in a network environment using logical connections to one or more remote computers. The remote computers may be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and typically include many or all of the elements listed above, as well as other elements not so mentioned. The logical connections may include any method supported by the available communications media.

[0159] Throughout this specification, including the claims, the term "comprising" should be understood to be synonymous with "comprising at least one" unless otherwise specified. In addition, ranges defined in this specification, including the claims, should be understood to be inclusive of their final values ​​unless otherwise specified. The specific values ​​of the elements described should be understood to be within acceptable manufacturing or industrial tolerances known to those skilled in the art, and the use of the terms "substantially" and / or "approximately" and / or "generally" should be understood to mean falling within such tolerances.

[0160] The terms "record" and "receive" may be used interchangeably throughout this disclosure unless otherwise indicated.

[0161] Although the present disclosure has been described herein with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure.

[0162] It is intended that the specification and examples be considered as exemplary only, with the true scope of the disclosure being indicated by the following claims.

[0163] Reference herein to a patent document or other matter identified as prior art is not to be construed as an admission that the document or other matter was publicly known at the priority date of any of the claims or that the information contained in the document or other matter was part of the common general knowledge.

Claims

1. 1. A computer-implemented method for improving a segmentation mask, comprising: Providing a first segmentation mask associated with the image; providing user input comprising a user defined region of the image; and refining the first segmentation mask based on image data of the image using a computer-implemented algorithm to obtain a second segmentation mask; The algorithm is configured to penalize violations of the user input. Computer-implemented method.

2. the first segmentation mask is associated with a region of interest in the image; and / or the user-defined region of the image is adapted by the user to modify the first segmentation mask; and / or The user-defined regions include a first user-defined region adapted by the user to extend the first segmentation mask and / or a second user-defined region adapted by the user to restrict the first segmentation mask. The method of claim 1.

3. The algorithm comprises: modifying the first segmentation mask based on the user-defined region in a fuzzy manner by applying a penalty rule; and / or including the first user-defined region in the first segmentation mask in a fuzzy manner by applying a penalty rule; and / or Excluding the second user-defined region from the first segmentation mask in a fuzzy manner by applying a penalty rule. It is configured as follows: The method according to claim 1 or 2.

4. The algorithm comprises: impose penalties on violations of said user-defined domains; and / or Penalizing violations of extending the first segmentation mask by the first user-defined region; and / or penalizing violations of excluding the second user-defined region from the first segmentation mask; The penalty rules are structured to 4. The method according to claim 1 .

5. The user input further includes a user-defined confidence level associated with the user-defined region; The algorithm is configured to penalize violations of the user input as a function of the user-defined confidence level.

5. The method according to any one of claims 1 to 4.

6. the image data optionally including statistical measures of image features including intensity, color, motion, and / or texture; The image data is used by the algorithm to identify image regions to refine the first segmentation mask.

6. The method according to any one of claims 1 to 5.

7. The algorithm is configured to utilize local likelihood statistics and / or context distributions based on the image data to adjust the segmentation boundary of the first segmentation mask.

7. The method according to any one of claims 1 to 6.

8. The algorithm is based on a predefined function: [0010] where Φ is an implicit level set function, and optionally, either {x|Φ(x)<0} or {x|Φ(x)>0} represents the image region inside the first segmentation mask; the function D(Φ) represents a penalty term configured to penalize a reduced fit of the first segmentation mask compared to the image data; the function F(Φ) is configured to penalize violations of the user input; The function R(Φ) represents the regularization penalty term of the segmentation mask.

8. The method according to any one of claims 1 to 7.

9. The algorithm is configured to compare a local distribution around each of a plurality of image points with a distribution in image regions inside the first segmentation mask and with a distribution in image regions outside the first segmentation mask, such that points with similar contextual information are grouped together in the second segmentation mask.

9. The method according to any one of claims 1 to 8.

10. The penalty rule is the predefined function [0025] In the term [0030] is defined by, where: M ｉｎ is a mask of all points within the user defined region that the segmentation mask should span, M оｕｔ is a mask of all points outside the user-defined region to which the segmentation mask should be restricted, v is a map of user-defined penalty weights, P(s) is a monotonically non-decreasing function of the value s.

10. The method according to any one of claims 1 to 9.

11. The first segmentation mask is generated automatically by a computer-implemented algorithm.

11. The method according to any one of claims 1 to 10.

12. the first and / or second segmentation masks are displayed on a display device; and / or The first segmentation mask is displayed on a display device. the user input is received by an input device; the user input and the first segmentation mask are provided to a computing device. the computer-implemented algorithm being executed by the computing device to obtain the second segmentation mask. The second segmentation mask is displayed on the display device, optionally adjacent to or in place of the first segmentation mask.

12. The method according to any one of claims 1 to 11.

13. The second segmentation mask of iteration N is used as the first segmentation mask of iteration N+1.

13. The method according to any one of claims 1 to 12.

14. 1. A method for generating a training data set for an artificial intelligence (AI) algorithm, comprising: - applying the method according to any one of claims 1 to 13 to a plurality of first segmentation masks to obtain a plurality of second segmentation masks; generating the training data set based on the plurality of second segmentation masks and optionally the plurality of first segmentation masks; A method comprising:

15. 1. A method for training an artificial intelligence (AI) algorithm, comprising: Executing the method of claim 14 to obtain a training data set; and training the artificial intelligence (AI) algorithm in a supervised manner using the training data set; The plurality of first segmentation masks are used as inputs during training, and the plurality of second segmentation masks are used as target outputs. method.

16. 1. A computing device comprising: At least one processor; and at least one memory storing computer-executable instructions which, when executed by the processor, cause the computing device to perform the method of any one of claims 1 to 15. Computing device.

17. configured to be associated with a display device such that the first and / or second segmentation masks are displayed on the display device; and / or configured to be associated with an input device configured to receive and provide the user input to the computing device; 17. The computing device of claim 16.

Images