Method for fine segmentation of mask

Abstract

The invention relates to a method for fine segmentation of a mask. The method comprises the following steps: providing a first segmentation mask associated with a region of interest in an image; providing a user input comprising a user-defined area of the image; the first segmentation mask is refined based on image data of the image by using a computer-implemented algorithm configured to penalize violations to the user input to obtain a second segmentation mask.

Description

Technical Field

[0001] The present invention relates to the technical field of segmentation masks for image segmentation. Background Art

[0002] Image segmentation has a variety of functions for processing image data, for example in the context of image detection, image content detection or image classification.

[0003] In medical imaging methods, image segmentation can be used, for example, for automatic or semi-automatic lesion or ROI (region of interest) detection. Examples of medical imaging methods include ultrasound imaging, mammography, and tomosynthesis, among others.

[0004] Computer-aided image segmentation has become important for fields such as cardiology, oncology, radiology, and other medical fields. With the advent of deep learning and other artificial intelligence (AI) techniques, the ability of computers to perform visual tasks such as recognition, localization, and generation has become popular. For example, Convolutional Neural Networks (CNNs) have been shown to be particularly effective for image segmentation tasks and can therefore be used to solve lesion segmentation tasks in the medical field and other fields.

[0005] However, such techniques may not be perfect and therefore an examiner, such as a physician, physicist, or radiologist, may wish or be required to review and, in certain circumstances, adjust the results based on their skills and experience.

[0006] The conventional solution of manual post-processing of the automatically predicted segmentation masks is rather cumbersome and unintuitive, ultimately resulting in prolonged time and effort.

[0007] In view of the foregoing, there exists a need to address these drawbacks and related disadvantages. Summary of the invention

[0008] A computer-implemented method for refining a segmentation mask is provided, comprising:

[0009] providing a first segmentation mask associated with the image;

[0010] providing a user input, the user input comprising a user-defined region of the image;

[0011] The first segmentation mask is refined based on the image data of the image by using a computer-implemented algorithm to obtain a second segmentation mask, wherein the algorithm is configured to penalize violations of the user input.

[0012] By providing this method, it is possible to refine the first segmentation mask based on user prompts, i.e. user input. For example, the user may input hints about regions in a vague manner to include or remove one or more regions in the first segmentation mask. Obtaining the second segmentation mask may be automatically guided by the user prompts. According to the present method, the first segmentation mask may become user adaptive. Therefore, the adjustment or refinement of the first segmentation mask may be considered to be at least semi-automatic, because the user only needs to provide vague user prompts, rather than precise changes to the mask, which advantageously saves his time and effort.

[0013] User inputs may thus also be referred to as user constraints.

[0014] The first segmentation mask may, for example, be automatically generated by a computer-implemented algorithm, such as based on an AI-based algorithm.

[0015] The first segmentation mask may be associated with a region of interest in the image. The region of interest may be associated with only a portion of the image. Thus, the first segmentation mask may, for example, cover only a portion of the image, ie the region of interest (ROI).

[0016] The user-defined region of the image may be adjusted by the user to modify the first segmentation mask. In other words, the user-defined region of the image may be determined and / or adjusted by the user to modify the first segmentation mask.

[0017] Thus, modifying the segmentation mask may mean expanding the first segmentation mask and / or shrinking the first segmentation mask.

[0018] For example, the user-defined region may include a first user-defined region adjusted by a user to extend the first segmentation mask.

[0019] In another example, the user-defined region may include a second user-defined region adjusted by a user to limit the first segmentation mask.

[0020] The algorithm can be configured to modify the first segmentation mask based on the user-defined area in a fuzzy manner by applying a penalty rule. The penalty rule can be predefined or defined or selected by the user or the system during the implementation of the method of the present disclosure. The penalty rule can also be preset and / or predefined and can be adjusted or can be adjusted during the implementation of the method of the present disclosure.

[0021] For example, the user defined region may be used as a soft criterion based on which the first segmentation mask may be modified.

[0022] For example, the algorithm may be configured to include the first user-defined region into the first segmentation mask in a fuzzy manner by applying a penalty rule.

[0023] In another example, the algorithm may be configured to remove the second user-defined region from the first segmentation mask in a fuzzy manner by applying a penalty rule.

[0024] The algorithm may include at least one penalty rule.

[0025] According to one example, penalty rules may be configured to penalize violations of user-defined regions.

[0026] According to another example, the penalty rule may be configured to penalize a violation of extending the first segmentation mask by the first user-defined region. In other words, the situation involves a situation where the first segmentation mask refined by the basis, such as the basis refinement based on the image data, does not match the first user-defined region. Therefore, the basis refinement based on the image data may be considered a violation of the user input. The violation may be penalized.

[0027] According to yet another example, the penalty rule may be configured to penalize a violation of removing the second user-defined region from the first segmentation mask. In other words, the situation involves a situation where the first segmentation mask refined by the basis, such as the basis refinement based on the image data, does not completely exclude the second user-defined region. Therefore, the basis refinement based on the image data may be considered as a violation of the user input. The violation may be penalized.

[0028] The user input may also include a user defined confidence level that may be associated with the user defined region. The algorithm may be configured to penalize violations of the user input according to the user defined confidence level.

[0029] For example, a parameter may be given in the method so that the confidence level of the user input can be adjusted, for example manually by the user. In other words, the user input may be processed in a fuzzy framework so that the confidence level can be adjusted in the automatic segmentation, as described below. This advantageously provides greater flexibility in terms of user error tolerance.

[0030] The image data may include statistical criteria for image features, such as low-level image features. The image features may include, for example, at least one of contour, edge, angle, intensity, color, motion and / or texture.

[0031] The image data may be used by an algorithm to differentiate image regions to refine the first segmentation mask.

[0032] The algorithm may be configured to utilize statistics based on contextual distribution and / or local likelihood of the image data to adjust the segmentation boundaries of the first segmentation mask.

[0033] The algorithm can be based on the following predefined functions:

[0034]

[0035] Φ can be an implicit level set function, where the image region within the first segmentation mask is optionally represented by {x|Φ(x)<0} (assumed in the equation examples below) or {x|Φ(x)>0}. "x" can be an image point, such as a pixel, voxel, or a point of any dimension.

[0036] The function D(Φ) may represent a penalty term configured to penalize a reduced level of fit of the first segmentation mask compared to the image data. Thus, the function D(Φ) may represent a penalty for a poor fit of the first segmentation mask compared to the image data.

[0037] The function F(Φ) may be configured to penalize a violation of the user input. Thus, the function F(Φ) may, for example, represent a fuzzy penalty for a violation of the user input by segmentation.

[0038] The function R(Φ) can represent the regularization penalty term of the segmentation mask.

[0039] An algorithm, such as the function D(Φ), may be configured to compare a local distribution surrounding each of a plurality of image points with distributions of image areas within and outside a first segmentation mask so that points with similar contextual information may be grouped together in a second segmentation mask.

[0040] For example, the algorithm may use an optimal transmission metric (eg, the Weisserstein metric) to compare the distributions. For example, the optimal transmission metric may be defined by the function D(Φ).

[0041] For example, in the predefined function The optimal transmission metric is defined by the following terms, and / or the function D(Φ) may include the following terms:

[0042] D(Φ)=α1∫ Ω H(Φ)W1(p(x),p out )dx+α1∫ Ω (1-H(Φ))W1(p(x),p in )dx (2)

[0043] in:

[0044] Ω can be the domain where segmentation is performed, such as the entire region of the image;

[0045] α1≥0 may be a weight of the distribution similarity term, such as a predefined weight;

[0046] H(Φ) can be a Heaviside function;

[0047] W1 can be an optimal transport metric, such as the 1-Weisserstein metric;

[0048] p(x) can be a local distribution about a point x (e.g., a pixel, a voxel, or a point of any dimension);

[0049] p out It can be the distribution of pixels outside the segmentation mask;

[0050] p in It can be the distribution of points within the segmentation mask.

[0051] An image point may include an image pixel, an image voxel, or an image point of any dimension.

[0052] An algorithm, such as the function D(Φ), may be configured to take into account information of local likelihoods so that likelihoods of points from image regions within the first segmentation mask and from image regions outside the first segmentation mask may compete, which separates points within the image region from points outside the image region.

[0053] For example, the function D(Φ) may include the following terms:

[0054] D(Φ)=-α2∫ Ω H(Φ)logp out (I(x))dx-α2∫ Ω (1-H(Φ))logp in (I(x))dx (3)

[0055] in:

[0056] α2≥0 may be a weight of the likelihood term, such as a predefined weight;

[0057] I(x) can be the value of point x in image I.

[0058] The penalty rule can be, for example, in a predefined function It is defined by the following items:

[0059] F(Φ)=∫ Ω M in (x)P(ν·Φ)dx+∫ Ω M out (x)P(-ν·Φ)dx (5)

[0060] in:

[0061] M in can be a mask of all points within a user-defined region to which the segmentation mask should extend;

[0062] M out can be a mask of all points outside a user-defined region that the segmentation mask should be restricted to avoid;

[0063] v(x) can be a user-defined penalty weight map, i.e., a map with a user-defined confidence level. For example, v(x) can be a function (hence a map) of the position x in the image and / or segmentation mask. For example, the user can include constraints (i.e., mask M in ) is more confident, and the removal of the restriction condition (i.e., the mask M out ) is less confident. For this reason, a confidence level ν(x) can be associated with each position.

[0064] P(s) may be a monotonic non-decreasing function according to the input value s. Thus, the input value s may be used to refer to, for example, an input value of the function P. In one example, the input value s may be a value of (ν·Φ) or (-ν·Φ).

[0065] For example, an algorithm such as the function R(Φ) may be configured to smooth the segmentation boundaries of the second segmentation mask. The function R(Φ) may be, for example, a predefined function It is defined by the following items:

[0066]

[0067] in:

[0068] λ>0 can be the weight of the regularization term.

[0069] So, according to a specific example, the predefined function It can be defined as follows:

[0070]

[0071] The first segmentation mask may be automatically generated by a computer-implemented algorithm.

[0072] The image may be a medical image and / or an image obtained by a medical imaging system, optionally an ultrasound imaging system.

[0073] According to one or more examples, the first and / or second segmentation masks may be displayed on a display device.

[0074] The first segmentation mask may be displayed on a display device.

[0075] User input may be received through input devices.

[0076] The user input and the first segmentation mask may be provided to a computing device.

[0077] The computer-implemented algorithm may be executed by the computing device to obtain a second segmentation mask.

[0078] The second segmentation mask may be displayed on a display device, optionally adjacent to or instead of the first segmentation mask.

[0079] The method can be iterated. For example, the second segmentation mask of iteration N can be used as the first segmentation mask in iteration N+1.

[0080] The method and, for example, the computer-implemented algorithm may be performed at least partially in real time or pseudo real time. For example, obtaining the second segmentation mask by refining the first segmentation mask may be performed in real time or near real time so that a user providing user input does not perceive any significant delay between inputting the user input and obtaining the second segmentation mask.

[0081] The present disclosure may also relate to a method of generating a training data set for an artificial intelligence (AI) algorithm, comprising:

[0082] Applying the method as described above to a plurality of first segmentation masks to obtain a plurality of second segmentation masks;

[0083] Based on the plurality of second segmentation masks and optionally the plurality of first segmentation masks, the training dataset is generated.

[0084] The present disclosure may also relate to a method of training an artificial intelligence (AI) algorithm, comprising:

[0085] Execute the method as described above to obtain a training data set;

[0086] The artificial intelligence (AI) algorithm may be trained in a supervised manner using the training dataset, wherein the plurality of first segmentation masks may be used as input and the plurality of second segmentation masks may be used as target output during training.

[0087] Examples of artificial intelligence (AI) algorithms may include machine learning models, such as decision trees, random forests, logistic regression, gradient boosting, neural networks (such as convolutional neural networks (CNNs)), and / or support vector machines. Predefined algorithms can be used to perform classification and prediction tasks, such as image recognition. For example, a decision tree algorithm can be used to classify images of objects based on their features. Other examples of machine learning models may include deep learning models, such as neural networks, k-nearest neighbors, and / or clustering algorithms, which can be used for tasks such as anomaly detection and data segmentation. AI may refer to the development of computer systems and algorithms that can perform tasks that typically require human intelligence, such as visual perception. AI systems and algorithms can be trained using large data sets and sophisticated algorithms, which can enable them to recognize patterns and trends, make predictions, and optimize results.

[0088] The present disclosure may also relate to a computing device comprising:

[0089] at least one processor; and

[0090] At least one memory stores computer executable instructions, which, when executed by the processor, cause the computing device to implement the method as described above.

[0091] For example, the computing device may be configured to compute or receive from an external system a first segmentation mask associated with the image;

[0092] receiving user input, the user input comprising a user-defined region of the image;

[0093] The first segmentation mask is refined based on the image data of the image using a computer-implemented algorithm to obtain or calculate a second segmentation mask, wherein the algorithm is configured to penalize violations of the user input.

[0094] Computer-executable instructions may, for example, include computer-implemented algorithms.

[0095] The computing device may be configured to be associated with a display device such that the first and / or second segmentation masks may be displayed on the display device.

[0096] The computing device may be configured to be associated with an input device configured to receive user input and provide it to the computing device.

[0097] A processor (or processing unit) may be a component of an electronic device that may be responsible for performing computing tasks. A processing unit may be or may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), and / or an application specific integrated circuit (ASIC). The method according to the present disclosure may also be run on a virtual server.

[0098] The present disclosure may also relate to a system for refining a segmentation mask, the system comprising a method for performing any example according to the present disclosure. For example, the system may include or may be a computing device as described above.

[0099] The present disclosure may also relate to an imaging (eg ultrasound imaging) system comprising means for performing the method according to any example of the present disclosure.

[0100] The present disclosure may also relate to a computer program, the program comprising instructions, which, when the program is executed by a computer, cause the computer to perform the method according to any example of the present disclosure.

[0101] The present disclosure also relates to a computer-readable medium, the medium comprising instructions, which, when executed by a computer, cause the computer to perform a method according to any example of the present disclosure.

[0102] Unless otherwise contradicted, the elements described above and in this specification are intended to be combinable.

[0103] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are provided for the purpose of illustration only and are not restrictive of the disclosure as claimed.

[0104] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present disclosure and, together with the description, serve to support and explain the principles thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 A method of refining a segmentation mask according to an example of the present disclosure is schematically illustrated.

[0106] Figure 2 A method of obtaining a second segmentation mask according to an example of the present disclosure is shown.

[0107] Figure 3 A method of training an artificial intelligence (AI) algorithm according to an example of the present disclosure is shown.

[0108] Figure 4 A method of providing and processing user input according to an example of the present disclosure is shown.

[0109] Figure 5a Show according to Figure 1 The image is a schematic example of the operation S1 of the method.

[0110] Figure 5b Show according to Figure 1 illustrative example of a first segmentation mask of operations S2 and S3 of the method.

[0111] Figure 5c Show according to Figure 1 illustrative example of a first segmentation mask and a user input in operation S4 of the method.

[0112] Figure 5d Show according to Figure 1 illustrative examples of the second segmentation mask of operations S5 and S6 of the method.

[0113] Figure 6 A schematic diagram of an ultrasound system 10 is shown according to an example of the present disclosure. DETAILED DESCRIPTION

[0114] Reference will now be made in detail to examples of the present disclosure, examples of which are illustrated in the accompanying drawings. Throughout the drawings, the same reference numerals will be used wherever possible to refer to the same or like parts.

[0115] Figure 1The method of refining the segmentation mask according to an example of the present disclosure is schematically shown. The method may include multiple operations. One or more operations may be implemented, for example, by corresponding software modules.

[0116] In an optional first operation S1 , an image may be provided. Figure 5a An example of an image 300a is shown. The image may be provided, for example, by an imaging system, such as a medical imaging system. The image may be associated with a medium scanned by the imaging system. For example, the imaging system may include at least one of: mammography, tomosynthesis, magnetic resonance imaging (MRI), single photon emission computed tomography (SPECT) scanning, positron emission tomography (PET) scanning, optical coherence tomography (OCCT), optical coherence tomography (OCT), X-ray examination, and ultrasound imaging. Figure 6 An exemplary ultrasound system is shown in FIG. 1 . In general, an image may be obtained by performing Figure 1 The image may be obtained by the first system of the method, or the image may be prepared or obtained by the second system and provided to the first system.

[0117] In operation S2, a first segmentation mask is provided. For example, the first segmentation mask may be obtained in operation S2 based on the image data of the image of operation S1. The first segmentation mask may be associated with a region of interest ROI in the image. The ROI may be or may include a lesion in a medium, such as human or animal tissue.

[0118] In one example, a first segmentation mask may be obtained by detecting and segmenting a region of interest (ROI) in the medium. For example, a predefined computer-implemented segmentation algorithm may be used, such as an AI-based algorithm, such as, for example, a neural network, such as a convolutional neural network (CNN). The region of interest may be associated with only a portion of the image. Thus, the first segmentation mask may, for example, cover only a portion of the image, i.e., the region of interest (ROI).

[0119] However, it is also feasible that the image and / or the first segmentation mask are only obtained from a communication interface, a data storage or the like. For example, the image and / or the first segmentation mask may be predefined and / or pre-stored and provided to a system performing a method according to the present disclosure.

[0120] In an optional operation S3, a first segmentation mask may be displayed. For example, the first segmentation mask may be displayed on the image, such as in Figure 5b As shown in the example in .

[0121] In operation S4, user input is provided. The user input may include a user-defined area, such as Figure 5c A user-defined region of the image may be provided by the user to modify the first segmentation mask.

[0122] In operation S5, the first segmentation mask is refined based on the image data of the image using a computer implemented algorithm to obtain a second segmentation mask. The algorithm thus penalizes violations of the user input.

[0123] In optional operation S6, a second segmentation mask may be displayed. The second segmentation mask may be displayed on the display device, optionally adjacent to the first segmentation mask or replacing the first segmentation mask. Figure 5d is shown in the example.

[0124] In operation S7, the method may be iterated. For example, the second segmentation mask of iteration N may be used as the first segmentation mask in iteration N+1. Thus, the user may successively refine the segmentation mask by modifying the (first) segmentation mask in each iteration.

[0125] The method may be performed at least partially in real time. For example, obtaining the second segmentation mask in operation S5 may be performed in real time or nearly real time so that a user providing user input does not perceive any significant delay when focusing on an image displayed on a display device.

[0126] Figure 2 A method of obtaining a second segmentation mask according to an example of the present disclosure is shown. Figure 2 The method can correspond to or Figure 1 A specific example of operation S5 of the method. Figure 2 The method, especially operation S5, may be performed by a computer-implemented algorithm according to the present disclosure.

[0127] Operation S5 may include a base refinement operation S5a1 , in which the first segmentation mask is base refined, for example based on the image data.

[0128] The image data may include statistical criteria for image features, such as low-level image features. The image features may include, for example, at least one of contour, edge, angle, intensity, color, motion, and / or texture. The image data may be used by the algorithm to distinguish image regions to refine the first segmentation mask.

[0129] Operation S5a1 may be performed using an algorithm or some rules defined by the algorithm, as described below. The algorithm may be configured to adjust the segmentation boundaries of the first segmentation mask using statistics based on contextual distribution and / or local likelihood of the image data. The algorithm may be configured to be able to adapt based on features of the medium scanned by the imaging system, such as the type of medium (e.g., an organ type such as liver or breast).

[0130] The algorithm can be based on the following predefined functions:

[0131]

[0132] Φ can be an implicit level set function, where the image region within the first segmentation mask is optionally represented by {x|Φ(x)<0} (assumed in the equation examples below) or {x|Φ(x)>0}. "x" can be an image point, such as a pixel, voxel, or a point of any dimension.

[0133] The function D(Φ) may represent a penalty term that is configured to penalize a reduced level of fit of the first segmentation mask compared to the image data. Thus, the function D(Φ) may represent a penalty for a poor fit of the first segmentation mask compared to the image data. Thus, in the case of a reduced level of fit (i.e., a relatively poor fit), the penalty according to the function D(Φ) may increase.

[0134] The function F(Φ) may be configured to penalize a violation of the user input. Thus, the function F(Φ) may, for example, represent a fuzzy penalty for a violation of the user input by segmentation.

[0135] The function R(Φ) can represent the regularization penalty term of the segmentation mask.

[0136] An algorithm, such as the function D(Φ), may be configured to compare a local distribution surrounding each of a plurality of image points with distributions of image areas within and outside a first segmentation mask so that points with similar contextual information may be grouped together in a second segmentation mask.

[0137] For example, the algorithm may use an optimal transmission metric (eg, the Weisserstein metric) to compare the distributions. For example, the optimal transmission metric may be defined by the function D(Φ).

[0138] In one example, the predefined function The optimal transmission metric is defined by the following terms, and / or the function D(Φ) may include the following terms:

[0139] D(Φ)=α1∫ Ω H(Φ)W1(p(x),p out )dx+α1∫ Ω (1-H(Φ))W1(p(x),p in )dx (2)

[0140] in:

[0141] α1≥0 can be the weight of the distribution similarity term, and Ω can be the domain where the segmentation is performed, such as the entire area of ​​the image;

[0142] H(Φ) can be a Heaviside function;

[0143] W1 can be an optimal transport metric, such as the 1-Weisserstein metric;

[0144] p(x) can be a local distribution about a point x (e.g., a pixel, a voxel, or a point of any dimension);

[0145] p out It can be the distribution of pixels outside the segmentation mask;

[0146] p in It can be the distribution of points within the segmentation mask.

[0147] An image point may include an image pixel, an image voxel, or an image point of any dimension.

[0148] An algorithm, such as the function D(Φ), may be further configured to take into account information of local likelihoods so that there may be competition between likelihoods of points from image regions within the first segmentation mask and from image regions outside the first segmentation mask, which separates points within the image region from points outside the image region.

[0149] For example, the function D(Φ) may include the following terms:

[0150] D(Φ)=-α2∫ Ω H(Φ)logp out (I(x))dx-α2∫ Ω (1-H(Φ))logp in (I(x))dx (3)

[0151] in:

[0152] α2≥0 can be the weight of the likelihood term;

[0153] I(x) may be an image.

[0154] For example, an algorithm such as the function R(Φ) may be configured to smooth the segmentation boundaries of the second segmentation mask. The function R(Φ) may be, for example, a predefined function It is defined by the following items:

[0155]

[0156] in:

[0157] λ>0 can be the weight of the regularization term.

[0158] Operation S5 and / or S5a1 includes operation S5a2 for penalizing the violation of the user input by a basic refinement operation.

[0159] For example, the algorithm may include penalty rules. The penalty rules may be configured to penalize violations of user-defined areas.

[0160] The user-defined region may be used, for example, as a soft criterion based on which the first segmentation mask may be modified. For example, the algorithm may be configured to include the first user-defined region into the first segmentation mask in a fuzzy manner by applying a predefined penalty rule. In another example, the algorithm may be configured to remove the second user-defined region from the first segmentation mask in a fuzzy manner by applying a penalty rule.

[0161] The penalty rule can be, for example, in a predefined function It is defined by the following items:

[0162] F(Φ)=∫ Ω M in (x)P(ν·Φ)dx+∫ Ω M out (x)P(-ν·Φ)dx (5)

[0163] in:

[0164] M in can be a mask of all points within a user-defined region to which the segmentation mask should extend;

[0165] M out can be a mask of all points outside a user-defined region that the segmentation mask should be restricted to avoid;

[0166] v(x) can be a user-defined penalty weight mapping, i.e., a mapping of user-defined confidence levels;

[0167] P(s) may be a monotonically non-decreasing function of the input value s.

[0168] Thus, the computer-implemented algorithm may be configured to refine the first segmentation mask based on the image data of the image, see operation S5a1. However, in this case, the algorithm penalizes violations of the user input, i.e., restricts the base refinement based on the image data in view of the user input, see S5a2. Thus, in case the user-defined region does not match the base refinement, the base refinement is restricted to better respect the user-defined region.

[0169] Operations S5, S5a1 and / or S5a2 may include an optional operation S5a3 of taking into account a user defined confidence level.

[0170] For example, the user input may also include a user-defined confidence level that may be associated with the user-defined region.

[0171] The algorithm can be configured to penalize violations of user input according to a user-defined confidence level.

[0172] For example, a parameter may be given in the method so that the confidence level in the user input can be adjusted, for example manually by the user.

[0173] The penalty rule can be defined in a specific instance of item (5), for example, in the predefined function It is defined by the following items:

[0174] F(Φ)=∫ Ω M in (x)e v·Φ dx+∫ Ω M out (x)e -v·Φ dx (6)

[0175] in:

[0176] M in can be a mask of all points within a user-defined region to which the segmentation mask should extend;

[0177] M out can be a mask of all points outside a user-defined region that the segmentation mask should be restricted to avoid;

[0178] Figure 3 An exemplary method of training an artificial intelligence (AI) algorithm according to examples of the present disclosure is shown.

[0179] In operation S11, a set (ie, a plurality) of first segmentation masks may be provided. Optionally, the operation may include providing a plurality of images respectively associated with the plurality of first segmentation masks. The images are ideally different from each other, and / or the segmentation masks are ideally different from each other.

[0180] The first segmentation mask may correspond to Figure 1 Therefore, operation S11 may correspond to a plurality of operations S2 providing different first segmentation masks.

[0181] In operation S12, a plurality of second segmentation masks may be obtained. The second segmentation masks may correspond to Figure 1 Therefore, operation S12 may correspond to a plurality of operations S5 providing different second segmentation masks. In other words, the second segmentation mask may be obtained using the method according to the present disclosure.

[0182] In operation S13, a training data set may be generated. The training data set may include a set of samples. Each sample may include a first segmentation mask and / or an associated second segmentation mask, optionally together with an associated image.

[0183] In operation S14, the training data set may be used to train an artificial intelligence (AI) algorithm. Examples of artificial intelligence (AI) algorithms may include machine learning models such as decision trees, random forests, logistic regression, gradient boosting, neural networks (such as convolutional neural networks (CNNs)), and / or support vector machines.

[0184] For example, an AI algorithm may be trained in a supervised manner using a first segmentation mask and / or a corresponding image as input and a corresponding second segmentation mask as a target output.

[0185] In this way, the AI ​​algorithm can learn to predict the second refined segmentation mask based on the image and / or the first segmentation mask.The trained AI algorithm can thereby automate or at least partially automate the manual work of providing user input.

[0186] Figure 4 A method of providing and processing user input according to an example of the present disclosure is shown.

[0187] Figure 4 The method can correspond to Figure 1 A specific example of operation S4 of the method. Figure 1 The method, in particular operation S4, may be performed by a software module configured to receive and optionally process user input.

[0188] Operation S4 includes an operation S4a of providing a user-defined area of ​​the image. For example, the user-defined area may be provided by a user manually drawing the area using a user interface device associated with the computing device on which the method is executed. The user interface device may be, for example, a touch screen device (e.g. Figure 6 The control panel display shown in 49).

[0189] In operation S4b, it may be determined whether the user-defined area is intended to be an extension or a restriction of the first segmentation mask. For example, the method or corresponding software module may be associated with an additional input device that can be used by the user to select the type of the user-defined area. The additional input device may be, for example, one or more physical or software-implemented keys of a user interface device.

[0190] It is also possible to automatically detect the type of user-defined region by the method. For example, in the case where more than 50% of the user-defined region is outside the first segmentation mask (see e.g. Figure 5c In the case where more than 50% of the user defined area is within the first segmentation mask (see, e.g., Figure 5c As shown in the user defined area 304b), it can be determined that the user defined area is intended to be removed from the first segmentation mask.

[0191] Therefore, in operation S4b(o.1), i.e., option 1 of operation S4b, the user-defined region may be marked as a first user-defined region intended to be added to the first segmentation mask. In addition, in operation S4b(o.2), i.e., option 2 of operation S4b, the user-defined region may be marked as a second user-defined region intended to be removed from the first segmentation mask.

[0192] Moreover, operation S4 may include providing a user-defined confidence level for each provided user-defined area. For example, the user-defined confidence level may be associated with the user-defined area. In one example, the user-defined confidence level may be a number between 0 and 100, which may be manually input by a user using a user interface device.

[0193] The user defined regions, in particular the first and / or the second user defined regions, optionally together with one or more confidence levels, may be forwarded to operation S5 , where a second segmentation mask is obtained, or its corresponding algorithm as described above.

[0194] The following reference Figure 5a to Figure 5d The example explains in more detail Figure 1 , Figure 2 and Figure 4 method.

[0195] Figure 5a Show according to Figure 1 The image is a schematic example of the operation S1 of the method.

[0196] As mentioned above Figure 1 As described in operation S1 of FIG. 1 , the image 300 may be, for example, an ultrasound image, or any other type of image. In particular, the image may show a medium being scanned by ultrasound. The medium may be, for example, a part of a human body or an animal or a material. The image may be displayed on a display device, as described above and with reference to FIG. Figure 6 described.

[0197] The image may include a region of interest 302, such as a suspected lesion in the medium. In one example, the region of interest may be manually selected by a user by drawing a bounding box 301 that includes the region of interest. In other examples, the region of interest is automatically determined and detected.

[0198] Figure 5b Show according to Figure 1 illustrative example of a first segmentation mask of operations S2 and S3 of the method.

[0199] As mentioned above Figure 1 As described in operation S2 of , the first segmentation mask 303 may be automatically determined. The determination may also be based on a manually drawn bounding box 301.

[0200] The first segmentation mask 303 may be displayed together with the image 300, in particular by overlaying on the image 300. The first segmentation mask 303 may thereby indicate the edges of a region of interest, eg an envisaged lesion, muscle, organ or a part thereof in the scanned medium.

[0201] A user, for example an examiner such as a doctor or radiologist, may wish to review and in some cases based on his skills and experience adjust the first segmentation mask 303. In particular, the user may wish to modify the edges of the first segmentation mask by locally extending and / or restricting the mask. Figure 5c and Figure 5d An example of dealing with this problem is shown in .

[0202] Figure 5c Show according to Figure 1 illustrative example of a first segmentation mask and a user input in operation S4 of the method.

[0203] The present systems and methods may provide a means to allow user interaction to provide user defined regions that may be included in the ROI and regions that may be excluded.

[0204] In this example, the user provides two user defined areas 304a and 304b, for example by drawing lines on a display device and / or by setting one or more marker points 304c that define the edges of the user defined areas.

[0205] As reference Figure 4 As described, based on manual user selection and / or based on automated determination, user defined region 304a may be marked as a first user defined region intended to be added to the first segmentation mask. In addition, user defined region 304b may be marked as a second user defined region intended to be removed from the first segmentation mask.

[0206] In addition, as referenced Figure 4 As mentioned, the user may additionally input one or more user-defined confidence levels for one or each or some of the user-defined regions 304a and 304b.

[0207] Figure 5d Show according to Figure 1 illustrative examples of the second segmentation mask of operations S5 and S6 of the method.

[0208] Based on the image data of the image 300 , a second refined segmentation mask 305 is obtained using a computer-implemented algorithm, wherein the algorithm is configured to penalize violations of user input.

[0209] Therefore, an image region 305a is included in the second segmentation mask compared to the first segmentation mask 303. The added image region 305a is determined by an algorithm based on the image data, wherein the user input (ie the first user defined region 304a) may be used as a blurred user input.

[0210] Correspondingly, the image region 305b is removed from the second segmentation mask compared to the first segmentation mask 303. The removed image region 305b is determined by an algorithm based on the image data, wherein the user input (ie the second user defined region 304b) is used as the fuzzy user input.

[0211] It should be noted that the user input does not need to be very accurate. The statistical term of the algorithm according to the present disclosure can provide a more accurate segmentation boundary, while taking into account the user constraints in a fuzzy way.

[0212] Figure 6 A schematic diagram of an ultrasound system 10 is shown according to an example of the present disclosure.

[0213] The ultrasound system 10 may be, for example, a system for refining a segmentation mask according to the present disclosure. The system may, for example, be configured to implement the method of the present disclosure. However, the method of the present disclosure may also be implemented (at least partially) by an external system.

[0214] The ultrasound imaging system 10 may include:

[0215] - Probe 20;

[0216] - a processing unit 30 for processing images based on the signals received by the probe;

[0217] a control panel 40a connected to the processing unit, said control panel comprising at least keys 41 and a touch pad 42; and

[0218] - A display screen 50 for visualizing the image.

[0219] The probe 20 can be associated with the processing unit 30 via a cable 21 and / or via one or more wireless connections, and it is capable of transmitting ultrasonic waves W into the medium M and receiving ultrasonic waves W from the medium M, wherein the received ultrasonic waves are caused or generated by reflection of the transmitted ultrasonic waves on diffuse particles in the medium M.

[0220] The display screen 50 may be a screen for visualizing the image processed by the processing unit 30. The display device 50 may also visualize other information, such as the scale used in the image and / or configuration information for the processing or any information, such as help information or contextual gesture help for the touch pad 42.

[0221] The display screen may be hinged on the support arm 51 for better positioning for the user. The display screen is typically a large size (at least 20 inches) high resolution screen to better visualize the image for the user.

[0222] The control panel 40a is, for example, a part of the system housing 31, which includes a panel housing having a generally flat surface that is tilted toward the user so that the user can operate with one hand. The control panel 40a can be moved up and down by a stand to adapt to the height of the user, and can optionally be moved forward and backward to adapt to the user's position. The control panel 40a may include a control panel display 49 for visualizing a plurality of configuration information or any information dedicated to the user.

[0223] The processing unit 30 may be configured to refine the segmentation mask according to the present disclosure. The processing unit may also send data to an external device to implement the method of the present disclosure at least in part on the external device. Examples of external devices include servers, computers, dedicated workstations, devices for displaying images obtained from electronic control devices, or any other external devices. Therefore, the method according to the present disclosure may be performed by at least one of the processing unit or the external device. Moreover, the process for constructing image data (i.e., compiling ultrasound image data) based on the acquired data may be performed by the same processing device as the processing device used for the optimization process, or (at least in part) by another processing device.

[0224] According to other examples, the system 10 may include at least one processing unit (or processor) and memory. In some examples, the processor and the memory unit may be integrated in the system, or may be a computer or a computer linked to the system in communication. Depending on the exact configuration and type of the computing device, the memory (which stores instructions for evaluating ultrasound data or executing the methods described herein) may be volatile (e.g., RAM), non-volatile (e.g., RAM, flash memory, etc.), or some combination of the two. Moreover, the system 10 may also include storage devices (removable and / or non-removable), including but not limited to disks or optical disks or tapes. Similarly, the system 10 may also have one or more input devices, such as a keyboard, a mouse, a pen, voice input, etc., and / or one or more output devices, such as a display, a speaker, a printer, etc. One or more communication connections may also be included in the environment, such as 4G / 5G, WIFI, LAN, WAN, point-to-point, etc. In some embodiments, these connections may be operated to facilitate point-to-point communication, connection-oriented communication, connectionless communication, etc.

[0225] The system 10 may typically include some form of computer-readable media. Computer-readable media may be any available media that can be accessed by a processing unit (or processor) or other device including an operating environment. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media do not include communication media.

[0226] Communication media is implemented as computer readable instructions, data structures, program modules or other data in a modulated data signal (such as a carrier wave or other transport mechanism), and includes any information delivery media. The term "modulated data signal" refers to a signal whose characteristics can be set or changed to encode information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct wired connection and wireless media such as acoustic, RF, infrared, microwave and other wireless media. Any combination of the above should also be included in the scope of computer readable media.

[0227] System 10 can be a single computer running in a networked environment using logical connections to one or more remote computers. The remote computers can be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and typically include many or all of the above elements, as well as other elements that are not mentioned. The logical connections can include any method supported by the available communication media.

[0228] Throughout this specification, including the claims, the word "comprising" should be understood as being synonymous with "comprising at least one", unless otherwise specified. In addition, any ranges set forth in this specification, including the claims, should be understood to include their endpoints, unless otherwise specified. The specific values ​​of the described elements should be understood to be within acceptable manufacturing or industry tolerances known to those skilled in the art, and any use of the words "substantially" and / or "approximately" and / or "generally" should be understood to mean falling within such acceptable tolerances.

[0229] Throughout this disclosure, unless otherwise specified, the words "record" and "receive" may be used synonymously.

[0230] Although the present disclosure is described herein with reference to specific examples, it should be understood that these examples are merely illustrative of the principles and applications of the present disclosure.

[0231] It is intended that the specification and examples be considered as exemplary only, with the true scope of the disclosure being indicated by the following claims.

[0232] The reference herein to a patent document or any other material identified as prior art should not be taken as an admission that the document or other material was known or that the information it contained was part of the common general knowledge at the priority date of any of the claims.

Claims

1. A computer-implemented method for refining a segmentation mask, comprising: providing a first segmentation mask associated with the image; providing a user input, the user input comprising a user-defined region of the image; refining the first segmentation mask based on image data of the image by using a computer-implemented algorithm to obtain a second segmentation mask, Wherein the algorithm is configured to penalize violations of the user input.

2. The method of claim 1, wherein: The first segmentation mask is associated with a region of interest in the image; and / or, The user-defined region of the image is adjusted by the user to modify the first segmentation mask; and / or, The user-defined region includes a first user-defined region adjusted by the user to expand the first segmentation mask and / or a second user-defined region adjusted by the user to limit the first segmentation mask.

3. The method according to claim 1 or 2, wherein: The algorithm is configured to: modifying the first segmentation mask based on the user-defined region in a fuzzy manner by applying a penalty rule; and / or, including the first user-defined region into the first segmentation mask in a fuzzy manner by applying a penalty rule; and / or The second user-defined region is removed from the first segmentation mask in a fuzzy manner by applying a penalty rule.

4. A method as claimed in any one of the preceding claims, wherein: The algorithm includes at least one penalty rule, which is configured to: penalizing violations of said user defined area; and / or, penalizing violations of extending the first segmentation mask by the first user-defined region; and / or, Violations of removing the second user-defined region from the first segmentation mask are penalized.

5. A method as claimed in any one of the preceding claims, wherein: The user input also includes a user-defined confidence level associated with the user-defined region, wherein The algorithm is configured to penalize violations of the user input according to the user defined confidence level.

6. A method as claimed in any one of the preceding claims, wherein: The image data includes statistical criteria for image characteristics, the image characteristics optionally including intensity, color, motion and / or texture; and / or, The image data is used by the algorithm to distinguish image regions to refine the first segmentation mask.

7. A method as claimed in any one of the preceding claims, wherein: The algorithm is configured to adjust segmentation boundaries of the first segmentation mask using statistics based on context distribution and / or local likelihood of the image data.

8. A method as claimed in any one of the preceding claims, wherein: The algorithm is based on the following predefined functions: wherein Φ is an implicit level set function; wherein the image region within the first segmentation mask is optionally represented by {x|Φ(x)<0} or {x|Φ(x)>0}; in, The function D(Φ) represents a penalty term configured to penalize a reduced level of fit of the first segmentation mask compared to the image data; The function F(Φ) is configured to penalize violations of the user input; and The function R(Φ) represents the regularization penalty term of the segmentation mask.

9. A method as claimed in any one of the preceding claims, wherein: The algorithm is configured to compare a local distribution surrounding each of a plurality of image points with distributions of image areas within the first segmentation mask and outside the first segmentation mask so that points having similar contextual information are grouped together in the second segmentation mask.

10. A method as claimed in any one of the preceding claims, wherein: The penalty rule is in the predefined function It is defined by the following items: F(Φ)=∫ Ω M in (x)P(ν·Φ)dx+∫ Ω M out (x)P(-ν·Φ)dx (5), in: M in is the mask of all points within the user-defined region to which the segmentation mask should be extended; M out is a mask of all points outside the user-defined region that the segmentation mask should be restricted to avoid; v is a map of user-defined penalty weights; P(s) is a monotonically non-decreasing function of the value s.

11. A method as claimed in any one of the preceding claims, wherein: The first segmentation mask is automatically generated by a computer-implemented algorithm.

12. A method as claimed in any one of the preceding claims, wherein: displaying the first and / or second segmentation mask on a display device; and / or where: displaying the first segmentation mask on a display device; receiving the user input via an input device; providing the user input and the first segmentation mask to a computing device; The computer-implemented algorithm is executed by the computing device to obtain the second segmentation mask; and The second segmentation mask is displayed on the display device, optionally adjacent to or instead of the first segmentation mask.

13. A method as claimed in any one of the preceding claims, wherein: The method is iterated, wherein the second segmentation mask of iteration N is used as the first segmentation mask in iteration N+1.

14. A method for generating a training data set for an artificial intelligence (AI) algorithm, comprising: Applying the method of any one of the preceding claims to a plurality of first segmentation masks to obtain a plurality of second segmentation masks; Based on the plurality of second segmentation masks and optionally the plurality of first segmentation masks, the training dataset is generated.

15. A method for training an artificial intelligence (AI) algorithm, comprising: Implementing the method as claimed in claim 14 to obtain a training data set; training the artificial intelligence (AI) algorithm in a supervised manner using the training dataset; The plurality of first segmentation masks are used as input during training, and the plurality of second segmentation masks are used as target output.

16. A computing device comprising: at least one processor; and At least one memory storing computer executable instructions which, when executed by the processor, cause the computing device to perform the method of any of the preceding claims.

17. The computing device of claim 16, It is configured to be associated with a display device so that the first and / or second segmentation mask is displayed on the display device; and / or, It is configured to be associated with an input device configured to receive the user input and provide it to the computing device.

Images