Artificial intelligence assisted contouring in medical imaging

Abstract

The invention discloses contour drawing assisted by artificial intelligence in medical imaging. For AI-assisted segmentation, when the user changes the segmentation of the object, an undesired indication (e.g., samples from where the segmentation boundary has been present but no longer present due to the change) is used to notify the segmentation of the unchanged region. Negative and positive samples from user changes may be extracted. Information from these negative samples (with or without positive samples) is used by a machine learning model to re-segment the object. The user changes are used to correct the segmentation, even for regions that are not changed, minimizing the amount of manual adjustments required, and providing user-and / or purpose-specific segmentation.

Description

Technical Field

[0001] This embodiment relates to segmentation in medical imaging. Structural contouring is a fundamental task in medical imaging. This segmentation allows the user to focus on a region of interest and / or measure various properties of the structure. Segmentation aids in diagnosis and treatment planning. Background Art

[0002] Clinicians typically perform contouring manually on arbitrarily selected structures. Contouring of structures depends on the task and the user. Taking the contouring of the left atrial appendage (LAA) in intracardiac echocardiography (ICE) as an example, in a diagnostic setting, the sonographer may draw the contour closely to the clear image boundary of the LAA. In an interventional setting, for safety purposes, the physician may prefer to contour the structure in a way that includes more surrounding tissue to account for potential structural deformation over time (cardiac cycle or time from device manufacturing to implantation). A given model or process used for segmentation may not segment differently for different purposes. The user can then manually change the segmentation in a time-consuming and user-dependent manner.

[0003] In the field of interactive segmentation, the user adjusts the contours. Artificial intelligence (AI) has shown promising performance in image segmentation. AI may be helpful for interactive segmentation. The user input is transformed into an image map, such as a Gaussian heat map, a Euclidean distance map, a geodesic distance map, or simply a small constant disk. These maps are fed along with the initial image to generate a segmentation based on the user's changes. As the user makes more edits to modify the image, the same type of map is generated and used to generate an updated segmentation. These methods focus only on corrections, but ignore information about why the user made the corrections. Summary of the Invention

[0004] Systems, methods, and non-transitory computer-readable media for AI-assisted segmentation are provided. When a user changes the segmentation of an object, unexpected indications (e.g., samples from where a segmentation boundary once existed but no longer exists due to the change) are used to inform the segmentation of unchanged areas. Negative and positive samples from the user changes can be extracted. Information from these negative samples (with or without positive samples) is used by a machine learning model to re-segment the object. User changes are used to correct the segmentation (even for unchanged areas), minimizing the amount of manual adjustments required and providing segmentation that is specific to the user and / or purpose.

[0005] In a first aspect, a method for assisting segmentation in a medical imaging system is provided. An ultrasound image of a patient is acquired. An object in the ultrasound image is segmented, thereby generating a first outline of the object. A change to a portion of the first outline of the object is received from a user input. Positive and negative samples are extracted from the ultrasound image based on the changed outline. The object in the ultrasound image is segmented by inputting the ultrasound image and information about the positive and negative samples into a first machine learning model. Segmentation generates a second outline, and an image of the second outline is generated.

[0006] In a second aspect, a medical system for interactive segmentation is provided. A memory is configured to store a medical image. A user input is configured to receive a change from a user. An image processor is configured to receive, from the user input, a change from a first position to a second position for a first segmentation of the medical image, and to use a first patch for the first position and a second patch for the second position for a second segmentation of the medical image. The second segmentation is formed by a machine learning network. A display is configured to display a segmented image of the second segmentation.

[0007] In a third aspect, a method for AI-assisted contouring in a medical imaging system is provided. A graph is generated. The graph represents (a) the similarity of each location in the medical image to a first location of a first contour of an object in the medical image after a change is made to the first contour, and (b) the dissimilarity of each location to a second location of the first contour of the object in the medical image before the change. In response to an input of the medical image and the graph provided to a machine learning model, the machine learning model segments the object. The segmented object is displayed.

[0008] The illustrative embodiments listed below summarize additional features or aspects. Any one or more aspects described above or in the exemplary embodiments may be used alone or in combination with other illustrative embodiments, features, or aspects. Any aspect or feature of one of the methods, systems, or computer-readable media may be used with other aspects or features of the methods, systems, or computer-readable media. These and other aspects, features, and advantages will become apparent from the following detailed description of the preferred embodiments, which is to be read in conjunction with the accompanying drawings. The invention is defined by the following claims, and nothing in this section should be construed as limiting those claims. Additional aspects and advantages of the invention are discussed below in conjunction with the preferred embodiments and may be claimed later, independently or in combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The components and drawings are not necessarily drawn to scale, emphasis instead being placed upon illustrating the principles of the embodiments. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the different views.

[0010] Figure 1is a flow chart of one embodiment of a method for AI-assisted contouring in medical imaging;

[0011] Figure 2 is a flow chart of another embodiment of a method for AI-assisted contour drawing;

[0012] Figure 3 is a flow chart of one embodiment of using positive and negative samples from user changes for segmentation by AI;

[0013] Figure 4 illustrates changes to the profile made by the user;

[0014] Figure 5 illustrates the progression of segmentation based on manual and AI-assisted changes; and

[0015] Figure 6 is a block diagram of one embodiment of a medical system for AI-assisted interactive contouring. DETAILED DESCRIPTION

[0016] Given an initial outline (e.g., generated by an AI), the user only needs to adjust the outline locally. The adjusted negative samples (with or without positive samples) reflect the user's intention for the rest of the outline. The AI ​​or another AI uses these samples to understand the user's intention and adjust the entire outline globally.

[0017] In the example used in this article, interactive contouring is provided for anatomical structures in three-dimensional (3D) or four-dimensional (4D) ultrasound. Starting with a single user click, the system first aims to generate an initial contour of the target using a spatiotemporal AI segmentation model. If the user disagrees, the user refines a local segment of the initial contour. The refinement model then generates positive and negative samples (image patches around the corrected segment) based on the initial contour and the manually corrected contour segment. A deep network model uses a matching method of positive and negative samples to efficiently screen and update the rest of the contour (e.g., using images and information from the changed sample to perform segmentation).

[0018] Use negative samples or both negative and positive samples to understand the user's intention and globally update the contour to match the intention Figure 1 Instead of focusing solely on encoding the segments as corrected (i.e., positive samples), the proposed method uses the user's edits (positive samples) and the reasons for the edits by looking at the negative samples. Leveraging the contrast weight maps from the positive and negative samples, the proposed method generates high attention (e.g., high weight scores in the contrast weight map) on patches that have high similarity to the user's intention to contour (positive samples) and generates low attention on patches that the user modified (negative samples).

[0019] Instead of feeding edited outlines and the original image through the segmentation network, relational graph encoding can be used. Relational graph encoding is based on positive and negative examples, which creates a challenge for the machine learning network to learn the user's intent. The network does not know which parts of the image it should focus on. By utilizing contrastive weight maps for machine training, the network is given explicit instructions on the image patches it should focus on and which patches it should not focus on. The resulting machine learning model is trained to use the positive and negative examples of any local changes in the segmentation to inform the global segmentation of the object.

[0020] Figure 1 This is a flow chart of one embodiment of a method for assisting segmentation in a medical imaging system. The method uses AI-assisted contouring. Negative samples (with or without positive samples) are extracted to alter the segmentation to reflect the user's intention regarding where the contour should or should not be. The AI ​​uses the samples and the medical image to perform another segmentation based on the user's intention.

[0021] Figure 1 The method is provided by medical systems (such as Figure 6 The image processing is performed by a medical scanner or image processor (or another medical system). For example, an ultrasound scanner acquires an image. The image processor segments the image, receives input from a user for making changes, extracts samples from the changes, and generates an updated segmented image. The image processor can generate an image. Based on the image created by the image processor, a display displays the image.

[0022] The method is performed in the order shown (e.g., from top to bottom or numerical order), but other orders may also be used. Additional, different, or fewer actions may be provided. For example, in the case where the acquired image includes a segmentation, action 120 is not provided, such as previously performed. As another example, action 152 is not used, such as in the case where the sample is directly input without mapping. In another example, action 160 is not performed, such as in the case where the refined segmentation is stored in the patient's medical record for later review. In yet another example, actions are performed to configure the scanning and / or use of the output segmentation information.

[0023] Figure 2 yes Figure 1 A flowchart of an implementation of a method in a context or workflow. Figure 1 and Figure 2 You can use Figure 1 Other context or workflow of the method. Figure 2In the process, the workflow includes interactive contouring of ultrasound images. Starting with a two-dimensional (2D) image, such as a 2D multi-planar reconstruction (MPR) from a 4D ultrasound sequence, in act 120, the AI ​​module generates an initial contour of the target. The editing loop then begins with an interactive update in act 200. In act 130, the user edits a segment of the contour to include more or less adjacent soft tissue. In act 140, the sample generation module generates positive and negative samples (e.g., image patches) around the edited area. The AI ​​module then takes these samples as input and, in act 150, globally generates an updated contour result based on ray tracing sampling of the image space in act 210. If the user performs more edits, the loop continues in act 220 until satisfied.

[0024] In act 100, a medical scanner (e.g., an ultrasound scanner or imaging system) or an image processor acquires a medical image representing a patient. The patient is scanned. The resulting image is a medical image. In other embodiments, the medical image of the patient is acquired from a memory or transmitted via a computer network.

[0025] The medical image represents any part of a patient, such as an organ, head, or torso of the patient. In one embodiment, the medical image is an ICE image acquired using an ICE transducer (e.g., a transducer on an ICE catheter that is scanned from within the patient's heart). The ultrasound image represents a region of the patient's heart, such as imaging the patient's ostium, chambers, and / or valves and / or devices within the patient's body. ICE delivers real-time, high-resolution visualization of cardiac structures and provides continuous monitoring of catheter positioning within the heart. Procedural complications, such as pericardial effusion or thrombosis, can be detected early using ICE. ICE exhibits extraordinary patient tolerance, reduces the duration of fluoroscopy, and eliminates the need for general anesthesia or a second operator. Other ultrasound imaging, such as transesophageal echocardiography (TEE), may be used. In other embodiments, the medical image is a magnetic resonance image, an ultrasound image, a computed tomography image, or an x-ray image.

[0026] Medical images and corresponding segmentations can be of any spatial extent, such as 2D or 3D. Temporal variations can be provided, such as segmentation over a sequence of images over one or more cardiac cycles. 2D examples will be used in this article. A static 2D image example will be used in this article.

[0027] A medical image is data arranged as pixels or voxels and may or may not be formatted for display. A medical image may be data acquired by scanning, with or without additional processing (e.g., scan conversion). A medical image may be data used to generate a displayed image representing a patient, or may be a displayed image.

[0028] In act 120, the image processor segments the object in the ultrasound image. Segmentation identifies boundaries, regions, and / or volumes corresponding to the object. A segmented object, such as a boundary, region, and / or volume, has a contour or outer edge (curve or surface). Pixels, voxels, or scan locations corresponding to the object are identified as segments. Segmentation produces contours representing the outer edges of the object or the boundary between the object and other objects.

[0029] Any one or more objects can be segmented. For example, for ICE, the heart's ostia, valves, or chambers are segmented. Soft tissue structures (e.g., organs or muscles), bones, instruments, or implants can be objects that are represented and segmented in medical images.

[0030] Segmentation is performed using any process or function, such as intensity thresholding with low-pass filtering. In one embodiment, segmentation is performed using full width at half maximum (FWHM) or intensity thresholding with various standard deviations. Alternatively, a user (e.g., a radiologist) or an image processor performs the segmentation. In another approach, a machine learning model, such as an encoder-decoder based neural network, outputs (one or more) segmentations in response to an input of a medical image. An image-to-image, U-Net, or encoder-decoder network trained to output segmentations in response to an input of medical image data can be applied. The machine-trained model (segmenter) generates values ​​for features in a hidden layer in response to the input of a medical image and uses the values ​​of the features to output a segmentation.

[0031] Segmentation outputs an identification of the location occupied by the object. A contour (e.g., a boundary, an area distribution, or a volume distribution) of the object is output as the segmentation. The output contour is based on or generated in response to an input or displayed medical image.

[0032] The segmentation is displayed to the user. Given user preferences and / or the purpose of the segmentation, the user can modify the segmentation. For example, the user visually identifies one or more locations where the contour covers the object excessively or insufficiently. Figure 4 An example is shown. The initial segmentation 400 is generally elliptical, but includes a notch or gap formed by the outline 420 in the portion 430. As another example, the segmentation of the LAA does not adhere closely to the border. Because the physician is taking a measurement or making a diagnosis, the outline should adhere closely to the LAA border. In yet another example, the segmentation of the LAA adheres closely to the border. Because the physician is planning or performing an interventional procedure, a more inclusive outline is desired to account for potential structural deformation.

[0033] In act 130, the image processor receives a change to the segmentation. The contour is altered. For example, only a portion of the contour is altered.

[0034] The changes can be automated, such as using another machine learning model to identify the changes. In one method of interacting with the user, the changes are received from the user through user input. The user changes the outline using clicks and drags or other user interface operations using user input (e.g., a mouse, trackball, touchpad, touch screen, and / or keyboard). Figure 4 In the example of FIG, a user selects a point or location 410A along an outline 420. The user then drags the point or location 410A to a desired point or location 410B. For only a portion 430 of the outline 420, the outline 420 is modified to segment based on this change. Using a physical model, or by relating the change of a point or area to another function of the adjacent portion of the outline 420 (i.e., within the portion 430), the outline can have a defined elasticity. Other portions of the outline 420 are not changed, such as outside the portion 430.

[0035] The user selects location 410A as a representative change. Instead of changing all desired locations, one or more representative changes are made. For this representative area, one or more locations of each point are shifted, such as inward or outward relative to the object. The rest of the outline is not changed (no change) due to the user input. Figure 5 An example is shown. The initial contour 500 from the segmentation of act 120 includes two notches 502, 504. The user selects a point in notch 504 to change the contour. After making the change, contour 510 results in notch 504 no longer existing, but notch 502 still existing. Although shown as notches, any combination of notches and / or protrusions may be present.

[0036] The change or modification indicates the user's intention for the current segmentation. In action 140, the image processor extracts positive samples and / or negative samples from the image based on the changed contour. The image processor uses information from the medical image based on the change to indicate the user's intention for the segmentation.

[0037] Extract samples from medical images (such as ultrasound images). Samples are patches of any size, such as 4×8, 8×8, or other sizes or shapes. Other samples can be obtained, such as sparse sampling. Samples can be directional, such as having a longer extent parallel or perpendicular to the contour.

[0038] Any number of samples may be extracted. For example, one or more positive samples and one or more negative samples may be extracted. The resolution of the medical image and / or the size of portion 430 may control or influence the number of samples. For example, a sample may be extracted at the image resolution for each point along the contour of the altered portion 430. A set number of samples may be obtained, such as by sampling along the contour after dividing the altered portion into a set number of segments.

[0039] Positive samples can be extracted. For the portion 430 of the altered contour 400, samples are extracted along the contour after the alteration. The user has modified the segmentation so that the contour follows the desired tissue. The samples extracted from the contour 420 after the alteration reflect the positive placement of the contour 400.

[0040] Extract negative samples. For portion 430 of the altered contour 400, samples are extracted along contour 420 before the alteration. The user modified the segmentation so that contour 400 does not follow undesirable tissue. The samples extracted from contour 400 for portion 430 before the alteration reflect the negative or undesirable placement of contour 400. Other undesirable or negative samples may be extracted, such as at tissue away from the center of the contour (e.g., positioned inward or outward from the contour relative to the object, but not along the contour after the alteration).

[0041] In one embodiment, a contour has two segments: the edited segment and the remaining portion. For the edited segment, image patches centered around the altered contour are extracted. These patches represent positive samples because the user intended to move the contour. Image patches centered around the original corresponding contour segment before editing are extracted. These patches represent negative samples because the user did not intend for the contour to be located above them.

[0042] The extracted samples represent the tissue location showing the contour rather than the sample of the contour. These samples can be used to assist in identifying the contour location in other parts of the contour. The ultrasound image can be sampled in any way to test the segment location. Figure 2 In one method shown in , radial sampling is used. The center of the initially segmented object is used as the center of the radial lines. The position of the contour along each radial line is found by sampling along the radial line. The extracted samples can be used to assist in identifying the position along the radial line corresponding to the radius. For the remaining segments that are not edited by the user, radial patches centered around the contour can be extracted. Positive and negative samples are used to identify, for each ray or radial line, which patch along the ray should be the appropriate patch on which to place the contour. Other sampling of medical images can be, for example, along a Cartesian grid.

[0043] In act 150, the image processor segments the object in the medical (e.g., ultrasound) image. Segmentation produces another outline of the same object, where the outline may, but is not necessarily, different from the previous outline from the segmentation of act 120. This subsequent segmentation of act 150 uses the medical image and information from the sample as input to create the outline. The outline is provided as a boundary, region, volume, or other designation of the location of the object as distinct from other locations.

[0044] Segments are for parts that have not been changed by the user. The entire contour or object can be segmented to provide contours for the changed parts and other parts. Information from the sample on the medical image is used to identify the contours of all or other parts of the object (segmentation), even without any user input specifically for those other parts. Figure 5 An example is shown where initial contour 500 is modified by the user, resulting in contour 510. In act 150, the image processor uses samples from the user modification and the medical image to segment the object again, resulting in contour 520. Contour 520 does not include gap 502 and does not include gap 504.

[0045] After the change in action 130, the same object will be segmented again. The samples from the change are used for this re-segmentation. The samples are either directly input to the segmentation process as information or used to derive additional information, which is directly input to the segmentation process. Action 152 shows one way (generating a graph) in which additional information for the input is derived to the segmentation process.

[0046] Any of the segmentation processes or methods discussed above for act 120 can be used for the segmentation of act 150. The same or different methods as those used in act 120 are used in act 150. In one embodiment, the segmentation of act 150 uses a machine learning model having inputs of the medical image and information from the sample. The segmentation is generated by the machine learning model in response to input on two or more input channels, one input channel for the medical image and another input channel for information from the sample related to the changes made by the user.

[0047] Machine learning models are previously trained using training data. The model is formed by an architecture that defines learnable parameters. The training data includes many samples of input and corresponding ground truth (e.g., correct output), such as expert-annotated segmentations formed from a database of patient scans.

[0048] In order to train the machine learning model, a machine learning model arrangement (architecture) is defined. Any machine learning model now known or developed later can be used. For example, an image-to-image network, U-Net, DenseNet, ResNet, or encoder-decoder network is used. Downsampling layers, convolutional layers, pooling layers, dropout layers, skip connections, upsampling layers, and / or other neural network layers can be used. Any architecture that receives image (spatial) information to output spatial (e.g., image) information can be used. This definition is configured or programmed by learning. The number of layers or units of the model, the type of learning, and other characteristics are controlled by the programmer or user. In other embodiments, one or more aspects (e.g., the number of nodes, the number of layers or units, or the type of learning) are defined and selected by the machine during learning. Training data, including many samples of input data and corresponding outputs, is used for training. The relationship between input and output is machine-learned.

[0049] The image processor or another processor machine trains the model. The training learns the values ​​of the learnable parameters (e.g., weights, connections, filter kernels, and / or other learnable parameters of the defined architecture). Deep or another machine learning algorithm may be used. The weights, connections, filter kernels, and / or other parameters are the features being learned. For example, a convolution kernel is a trained feature. Using training data, the values ​​of the learnable parameters of the model are adjusted and tested to determine the values ​​that result in the best estimate of the output given the input. Adam or another optimization algorithm is used for training.

[0050] During training, a loss is minimized. This loss is derived from comparing the model's output to the expected output (i.e., the ground truth). The loss function is an L1, L2, or other error function between the network's output and the ground truth of the training data. Other losses can be used. Using optimization, different values ​​of the learnable parameters are tested across different samples of the training data to minimize the loss (or maximize the reward).

[0051] In one embodiment, the machine learning model includes inputs of information from samples, including negative samples or samples from the contour before the user makes a change and not after the user makes the change. The inputs are the samples themselves or information derived from them. For example, in action 152, a map is derived from the samples. The map indicates the user's intention for the changed location and / or other locations. For example, the map compares the extracted samples with the medical image at various locations sampled throughout the medical image. Sampling can be limited to within a threshold distance (e.g., 10% or 10 mm) of the initial contour from action 120, or the entire image can be fully or sparsely sampled. The results of the comparison, such as similarity, are used to derive a contrast value or weight for each location. In a radial line sampling embodiment, various radial patches are sampled along each of the various radial lines. Each position along each of the radial lines is sampled. Sampling can be limited to within a threshold distance from the previous contour. A patch centered at each sample location is extracted or defined.

[0052] The radial or sample patch is compared with the negative sample. A greater similarity or smaller dissimilarity indicates that the contour should not be located there. A comparison with a positive sample can also be used. A greater similarity or smaller dissimilarity of the radial patch to the positive sample indicates that the contour should be located there. In the case of using multiple positive or negative samples, an average metric can be used. In the case of using both positive and negative samples, a combination of comparison results can be used. Any combination, such as a weighted difference or ratio, can be used. In action 152, the graph is generated based on the weight or relationship value of the sample position of the medical image to the positive sample and / or negative sample from the changed position. Different positions along or around the object are sampled and compared, and a spatial distribution graph of the similarity (dissimilarity) of the extracted negative and / or positive samples is generated. This graph can then be used as input to the machine learning model.

[0053] In one method for action 152, an image processor generates a graph representing (a) the similarity of each location in the medical image to the location of the object's contour in the medical image after the contour is changed (positive samples) and (b) the dissimilarity of each location to the location of the object's contour in the medical image before the change (negative samples). The contour changes made by the user to a portion of the contour are used to map other portions of the contour that the user did not change. The relationship graph is formed using positive samples and / or negative samples. When both positive and negative samples are used, a contrast weight map is based on the similarity of each location in the image to the positive samples and the dissimilarity to the negative samples. Any combination of similarity and dissimilarity measures can be used. Any measure of similarity or dissimilarity, such as the minimum sum of absolute differences or the mutual correlation coefficient, can be used. The contrast weight map is formed by the positive samples and the negative samples, and the machine learning model outputs the contour in response to the input of the image and the contrast weight map.

[0054] Figure 3 An example of using a sample for map generation in act 152 is shown. Embedded features 332, 334, 336 of the sample are used in comparison to form a contrast weight map 340. The edited segment or contour 300 is used to extract positive samples 322 as patches around the portion of the contour after editing in act 140. The portion of the contour before editing is used to extract negative samples 324 as patches around the contour before editing in act 140. In act 320, patches 326 are extracted from the remaining portion of the segmentation, such as within a threshold distance from a previous segmentation along a radial line and / or within a threshold distance from a sample of the medical image, including samples taken at a location away from the portion changed by the user.

[0055] This graph uses embedded features 332, 334, 336 of patches 322, 324, 326. Embedded features 332 derived from the positive sample (patch 322), embedded features 334 derived from the negative sample (patch 324), and embedded features 336 derived from the sample patch 326 are used to form graph 340. Embedded features 332, 334, 336 are formed or output by encoder 330 or another neural network. The image processor generates a latent representation based on the input patches 322, 324, 326. Patches 322, 324, 326 are input to encoder 330. In response to the input, encoder 330 generates latent representations 332, 334, 336 as fingerprints.

[0056] Encoder 330 is a convolutional neural network and a transformer. In other methods, other encoders formed by other neural networks can be used, such as a fully connected neural network or a convolutional neural network with a soft-max output layer or a multi-layer perceptron. Deep learning neural networks can be used for encoder 330.

[0057] The encoder 330 operates based on how it is trained. Different training results in different values ​​for the learnable parameters and, therefore, different operations. In one approach, the encoder 330 or another machine learning model used to generate a potential representation of the patches 322, 324, 326 is trained with a decoder (e.g., the encoder 330 is trained as part of a segmentation network). Alternatively, the encoder 330 is trained using many samples of the patches as training data, where the ground truth is binary (0 or 1) that is positive or negative. The encoder 330 is trained to distinguish between positive and negative input samples.

[0058] Once trained, features 332, 334, 336 are extracted from the layers of the encoder 330 before the output layer (e.g., before a softmax or multi-layer perceptron or before a decoder). For example, features from one or more layers after the input layer and before the output (i.e., binary 1 or 0) layer of the encoder 330 are used. Alternatively, the encoder 330 outputs a latent representation, such as that trained as part of an encoder-decoder. The latent representation formed by the encoder 330 is used as a feature vector 332, 334, 336. Features from one or more hidden layers of the encoder 330 are used. The same encoder 330 with the same learned parameter values ​​is used for each patch of each type. In an alternative approach, different encoders 330 are trained for patches of different types (e.g., positive, negative, and image samples).

[0059] Given positive and negative samples (patches 322 and 324) and sample (e.g., radial) patches 326 from an initial or previous segmentation, these patches 322, 324, 326 are fed into a feature encoder 330 with shared weights for the different types of patches 322, 324, 326. A deep feature embedding (feature vectors 332, 334, 326) is obtained for each patch.

[0060] In action 152, a contrast weight map 340 is constructed based on these features. For each sample location with a corresponding patch 326, a score is formed from the features of the patch 326, the features of the positive patch 322, and the features of the negative patch 324. The score encourages similarity between the sample patch 326 and the positive sample patch 322, while discouraging similarity between the sample patch 326 and the negative sample patch 324, but the similarity is based on the embedded features. The map 340 then helps the downstream machine learning model 360 (network) to focus more on areas with high scores and less on areas with low scores (or vice versa, depending on how the map is generated). The weight map 340 is concatenated with the original image 350, and together they are input to the segmentation network (machine learning model for segmentation 360), which outputs an updated segmentation 370.

[0061] In one embodiment, the contrast weight is a score that maximizes the similarity of the sample (e.g., radial) patch embedding 336 to the positive patch embedding 332 while minimizing the similarity of the sample (e.g., radial) patch embedding 336 to the negative patch embedding 334. Given positive samples p1, p2, ..., p m , negative samples n1, n2, ..., n s , and radial (sample) patches are r1, r2, ..., r T, these image patches are input to a feature encoder 330, such as a convolutional neural network, followed by a transformer encoder. The superscript f' denotes their feature embeddings 332, 334, 336, The contrast weight of the i-th radial patch 326 is defined as:

[0062]

[0063] Where τ is a temperature parameter. The numerator indicates the similarity of the radial patch (after encoding) to the positive sample, while the denominator indicates the dissimilarity of the radial patch to the negative sample. A weight factor or score is determined for each radial patch 326. Other functions can be used. The weights of the contrast weight map 340 are the spatial distribution of the sample patches 326, which instruct or inform the downstream segmentation network 360 to pay appropriate attention to the relevant parts of the image. The weights distinguish between possible positive and possible negative positions of the contour based on the changes made by the user. The weight map is formed by calculating the scores of all sample (e.g., radial) patches 326. For each of the different position samples from the image, a weight is formed based on maximizing the similarity of the position sample to the positive sample, which is inversely weighted by the similarity of the position sample to the negative sample.

[0064] The contrast weight map 340 and the image 350 are input to a machine learning model (segmentation network 360), which outputs contours in response. Based on the local changes made by the user, the segmentation is globally updated. The machine learning model 360 is trained using many samples of the input image and information from the extracted patches with the ground truth segmentation. Figure 2 As shown in , if the user desires to make more edits to the contour, the process is repeated until the user's final segmentation goal is reached.

[0065] exist Figure 1 In step 160, an image processor generates an image. The image shows a segmentation (contour). The segmentation can be a graphic, such as a boundary line or shape of an object. The segmentation can be a modification of the medical image, such as coloring or highlighting the segmented object. In one method, the segmentation graphic is overlaid on the medical image. In other methods, a representation of the object is extracted from the image based on the segmentation. The representation of the object is displayed separately as a segmentation.

[0066] The segmented object is displayed to the user, who can use the segmentation for measurement, diagnosis, treatment planning, or other purposes.

[0067] Figure 6One embodiment of a medical system for interactive segmentation is shown. The medical system includes a display 630, a user input 620, a memory 610, and an image processor 600. The display 630, the image processor 600, the user input 620, and / or the memory 610 may be part of a medical scanner 640 (e.g., an ultrasound scanner), a computer, a server, a workstation, or other system for image processing medical images scanned from a patient. A workstation or computer without the medical scanner 640 may be used as the medical system.

[0068] Additional, different, or fewer components may be provided. For example, a computer network may be included for remote image generation of locally captured image data or for performing local predictions based on remotely captured image data. The machine learning model 605 may be applied as a standalone application on a workstation or local device, or as a service deployed on a network (cloud) architecture. In another example, the medical scanner 640 may not be provided.

[0069] Images (e.g., ultrasound images), segmentations, user edits, learned parameter values, machine learning models 605, feature values ​​(e.g., potential representations), positive samples, negative samples, image samples, display images, and / or other information are stored in a non-transitory computer-readable memory, such as memory 610. Medical images (e.g., magnetic resonance images, ultrasound images, computed tomography images, or x-ray images) are stored in memory 610. Memory 610 is an external storage device, RAM, ROM, database, and / or local memory (e.g., a solid-state drive or hard drive). The same or different non-transitory computer-readable media can be used for instructions and other data. Memory 610 can be implemented using a database management system (DBMS) and reside on a memory such as a hard disk, RAM, or removable media. Alternatively, memory 610 is internal to processor 600 (e.g., a cache).

[0070] Instructions for implementing the training or application processes, methods, and / or techniques discussed herein are provided on a non-transitory computer-readable storage medium or memory, such as a cache, buffer, RAM, removable media, hard drive, or other computer-readable storage medium (e.g., memory 610). Computer-readable storage media include various types of volatile and non-volatile storage media. In response to one or more instruction sets stored in or on a computer-readable storage medium, the functions, actions, or tasks illustrated in the accompanying drawings or described herein are performed. The functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy, and may be performed by software, hardware, integrated circuits, firmware, microcode, and the like, operating alone or in combination.

[0071] In one embodiment, the instructions are stored on a removable media device for reading by a local or remote system. In other embodiments, the instructions are stored in a remote location for transmission over a computer network. In yet other embodiments, the instructions are stored within a given computer, CPU, GPU, or system. Because some of the system components and method steps depicted in the accompanying drawings can be implemented in software, the actual connections between the system components (or process steps) may vary depending on how the embodiment is programmed.

[0072] User input 620 includes a keyboard, buttons, sliders, dials, trackballs, mice, touchpads, touch screens, and / or another device for receiving user input in a user interface. The user input is configured by image processor 600 or another processor to receive changes from the user. The segmentation is displayed. The user interface is configured to allow the user to change the segmentation, such as changing a portion of the outline. The user changes the boundary positioning of the segmentation based on the received user input (such as by clicking and dragging the displayed outline). More than one change can be made.

[0073] The image processor 600 is a controller, a control processor, a general-purpose processor, a microprocessor, a tensor processor, a digital signal processor, a three-dimensional data processor, a graphics processing unit, an application-specific integrated circuit, a field-programmable gate array, an artificial intelligence processor, a digital circuit, an analog circuit, a combination of the above, or other devices now known or later developed for processing image data. The image processor 600 is a single device, multiple devices, or a network of devices. For more than one device, parallel or sequential processing division can be used. The different devices that make up the image processor 600 can perform different functions. In one embodiment, the image processor 600 is a control processor or other processor of the medical scanner 640. The image processor 600 operates according to stored instructions, hardware, and / or firmware and is configured by the stored instructions, hardware, and / or firmware to perform the various actions described herein.

[0074] The image processor 600 or another remote processor is configured to train a machine learning architecture. Based on a network architecture provided by a user or other source and training data (e.g., formed from a database of expert curation of patient scans and ground truth values), the image processor 600 learns to relate one or more input variables (e.g., an image or an image and information from a negative sample) to an output (e.g., segmenting or distinguishing positive and negative samples). The result of the training is (one or more) machine learning models 605, such as a model for generating segmentation in response to negative samples or information derived from these negative samples and an input of a medical image. Other machine learning models discussed herein (e.g., the encoder 330 or a segmentation network for initial segmentation without using information from user-altered samples) can be trained by the image processor 600 and / or stored in the memory 610.

[0075] Alternatively or additionally, the image processor 600 is configured to apply (one or more) machine learning models 605 (i.e., apply a segmentation model with a medical image input, apply an encoder to generate embedded features of the patch, and / or apply a segmentation model with input of medical information and information derived from positive and / or negative samples from user changes). The machine learning model 605 may include a machine learning network for global segmentation based on local user changes.

[0076] Image processor 600 is configured to receive a change from one location to another location for segmenting a medical image. The change is received from user input 620. Image processor 600 is configured to use the patch associated with the one location and the patch associated with the other location to perform another segmentation of the medical image. The other segmentation is formed by machine learning network or model 605. The patch associated with the one location is at the location of the boundary before the user changes it, and the patch associated with the other location is at the location of the boundary after the user changes it.

[0077] In one approach, the machine learning network or model 605 is an image-to-image network (e.g., a U-Net or encoder-decoder) with one input for a medical image and another input for a relationship graph that relates each location in the medical image to the similarity of a patch at one location (negative) to a patch at another location (positive).

[0078] In another approach, the image processor is configured to generate feature embeddings for patches (a patch for one location, a patch for another location, and patches sampled from different locations in the medical image). A relationship graph is formed with weights that maximize similarity of the feature embeddings of the patches at different sample locations to the feature embeddings of the patches from the altered contour, weighted by (e.g., divided by) similarity of the feature embeddings of the patches at different sample locations to the feature embeddings of the patches from the contour before the user altered the contour.

[0079] The image processor 600 is configured to generate an output, such as an image showing the segmentation. The contours of the object are displayed alone or together with other information, such as an overlay on the medical image or an image of the object extracted from the medical image based on the segmentation. The image processor 600 is configured to generate a visualization of the segmentation (contours).

[0080] The display 630 is a CRT, LCD, projector, plasma TV, printer, tablet computer, smart phone, or other display device now known or later developed for displaying output, such as segmented images (segmented images). The segmented images previously segmented, modified, and / or modified by the user can be displayed sequentially and / or together.

[0081] Medical scanner 640 is a diagnostic or therapeutic scanner (e.g., an ultrasound, computed tomography, x-ray, or magnetic resonance scanner). Scanner 640 operates according to one or more settings to scan patient 642, who is resting on a bed or table 644. These settings control the scan, including transmission, reception, reconstruction, and image processing. A scanning protocol is followed to generate data representing patient 642, such as an ICE ultrasound image representing the patient's cardiac region. Scanner 640 uses these settings to image patient 642. Scanner 640 generates a medical image representing the object to be segmented.

[0082] The following is a list of non-limiting illustrative embodiments disclosed herein.One set or type of illustrative embodiments (eg, method or system) may be provided within or combined with other sets or types of illustrative embodiments.

[0083] Illustrative Example 1. A method for segmentation assistance in a medical imaging system, the method comprising: acquiring an ultrasound image of a patient; segmenting an object in the ultrasound image, the segmentation generating a first outline of the object; receiving a change to a portion of the first outline of the object from a user input; extracting positive and negative samples from the ultrasound image based on the changed outline; segmenting the object in the ultrasound image, the segmentation generating a second outline by inputting the ultrasound image and information of the positive and negative samples into a first machine learning model; and generating an image of the second outline.

[0084] Illustrative embodiment 2. The method of illustrative embodiment 1, wherein acquiring comprises acquiring an ultrasound image representing a region of a heart of the patient using an intracardiac echocardiographic transducer, wherein the object comprises an ostium, a chamber, or a valve.

[0085] Illustrative embodiment 3. The method of any one of illustrative embodiments 1-2, wherein segmenting to produce the first contour comprises segmenting by a second machine learning model, the second machine learning model outputting the first contour in response to an input of the ultrasound image.

[0086] Illustrative Embodiment 4. The method of any of Illustrative Embodiments 1-3, wherein receiving comprises receiving a user input change that shifts a position of a point on the first contour away from the first contour, the portion comprising the position shifted by the change and adjacent positions, wherein other portions of the first contour are free of the change.

[0087] Illustrative embodiment 5. The method according to any one of illustrative embodiments 1-4, wherein extracting comprises extracting positive samples as one or more patches in the ultrasound image including the portion of the first contour after the change, and extracting negative samples as one or more patches in the ultrasound image after the change that are spaced apart from the first contour.

[0088] Illustrative embodiment 6. The method of any one of illustrative embodiments 1-5, wherein extracting comprises extracting negative samples as one or more patches in the ultrasound image including the portion of the first contour before the changing.

[0089] Illustrative embodiment 7. The method of any one of illustrative embodiments 1-6, wherein generating the segmentation of the second contour comprises segmenting by a first machine learning model comprising a U-Net.

[0090] Illustrative embodiment 8. The method of any one of illustrative embodiments 1-7, wherein segmenting to generate a second contour comprises forming a contrast weight map based on positive samples and negative samples, and the second contour is output by the first machine learning model in response to input of the ultrasound image and information, the information comprising the contrast weight map.

[0091] Illustrative embodiment 9. The method of illustrative embodiment 8, wherein forming comprises, for each of the different position samples from the ultrasound image, forming a weight based on maximizing a similarity of the position sample to a positive sample, the weight being inversely weighted by a similarity of the position sample to a negative sample.

[0092] Illustrative embodiment 10. The method of any of illustrative embodiments 8-9, wherein forming comprises forming a contrastive weight map based on embedded features, the embedded features derived by an encoder from the positive and negative samples.

[0093] Illustrative embodiment 11. The method of any one of illustrative embodiments 8-10, wherein the contrast weight map is based on the similarity of each location in the ultrasound image to the positive sample and the dissimilarity to the negative sample.

[0094] Illustrative Embodiment 12. The method of any one of Illustrative Embodiments 1-11, wherein the information comprises a relationship graph formed by positive samples and negative samples.

[0095] Illustrative Embodiment 13. A medical system for interactive segmentation, the medical system comprising: a memory configured to store a medical image; a user input configured to receive a change from a user; an image processor configured to receive a change from a first position to a second position for a first segmentation of the medical image from the user input, and use a first patch about the first position and a second patch about the second position for a second segmentation of the medical image, the second segmentation formed by a machine learning network; and a display configured to display a segmented image of the second segmentation.

[0096] Illustrative Embodiment 14. The medical system of Illustrative Embodiment 13, wherein the medical image comprises a magnetic resonance image, an ultrasound image, a computed tomography image, or an x-ray image.

[0097] Illustrative Embodiment 15. The medical system of any of Illustrative Embodiments 13-14, wherein the alteration is a change in the positioning of a boundary of the first segmentation, wherein the first patch is at the location of the boundary before the alteration and the second patch is at the location of the boundary after the alteration.

[0098] Illustrative Embodiment 16. The medical system of any of Illustrative Embodiments 13-15, wherein the machine learning network comprises an image-to-image network having a first input for the medical image and a second input for a relationship graph relating locations in the medical image and similarities to the first patch and the second patch.

[0099] Illustrative embodiment 17. The medical system of illustrative embodiment 16, wherein the image processor is configured to generate feature embeddings for the first patch, the second patch, and samples from different locations in the medical image, the relationship graph being formed by weights that maximize similarity of the feature embedding of the patch at the different location to the feature embedding of the second patch, the similarity being weighted by similarity of the feature embedding of the patch at the different location to the feature embedding of the first patch.

[0100] Illustrative Embodiment 18. A method for artificial intelligence-assisted contouring in a medical imaging system, the method comprising: generating a map representing: (a) a similarity of each location in a medical image to a first location of a first contour of an object in the medical image after a change is made to the first contour and (b) a dissimilarity of each location to a second location of the first contour of the object in the medical image before the change; segmenting the object by a machine learning model in response to input of the medical image and the map to the machine learning model; and displaying the segmented object.

[0101] Illustrative embodiment 19. The method of illustrative embodiment 18, wherein generating the map comprises generating the map using embedded features of the patch for each position, the first location, and the second location.

[0102] Illustrative embodiment 20. The method of any of illustrative embodiments 18-19, wherein the change to the first contour is made by a user to a portion of the first contour, wherein other portions of the first contour are not changed by the user, and wherein segmenting comprises segmenting the portion and the other portions without requiring user input relative to the other portions.

[0103] The various improvements described herein may be used together or separately. Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments and that various other changes and modifications may be made therein by those skilled in the art without departing from the scope or spirit of the invention.

Claims

1. A method for segmentation assistance in a medical imaging system, the method comprising: obtaining ultrasound images of the patient; segmenting an object in the ultrasound image, the segmentation generating a first outline of the object; receiving a change to a portion of the first outline of the object from a user input; Extracting positive and negative samples from ultrasound images based on the changed contours; Segmenting an object in the ultrasound image, wherein the segmentation generates a second contour by inputting the ultrasound image and information of positive and negative samples into a first machine learning model; and Generate an image of the second contour. 2 . The method of claim 1 , wherein acquiring comprises acquiring an ultrasound image representing a region of a heart of the patient using an intracardiac echocardiographic transducer, wherein the object comprises an ostium, a chamber, or a valve.

3. The method of claim 1, wherein segmenting to produce the first contour comprises segmenting by a second machine learning model, the second machine learning model outputting the first contour in response to an input of an ultrasound image.

4. The method of claim 1 , wherein receiving comprises receiving a user input change that moves a position of a point on the first contour away from the first contour, the portion comprising a position and an adjacent position shifted with the change, and wherein other portions of the first contour are free of the change.

5. The method according to claim 1, wherein the extracting comprises extracting positive samples as one or more patches in the ultrasound image including the portion of the first contour after the change, and extracting negative samples as one or more patches in the ultrasound image after the change that are spaced apart from the first contour. The method according to claim 1 , wherein extracting comprises extracting negative samples as one or more patches in the ultrasound image including the portion of the first contour before the change.

7. The method of claim 1, wherein generating the segmentation of the second contour comprises segmentation by a first machine learning model comprising a U-Net.

8. The method according to claim 1, wherein segmenting to generate a second contour includes forming a contrast weight map based on positive samples and negative samples, and the second contour is output by the first machine learning model in response to the input of the ultrasound image and information, wherein the information includes the contrast weight map.

9. The method of claim 8, wherein forming comprises, for each of the different position samples from the ultrasound image, forming a weight according to maximizing a similarity between the position sample and a positive sample, the weight being inversely weighted by a similarity between the position sample and a negative sample.

10. The method of claim 8, wherein forming comprises forming a contrastive weight map based on embedded features, the embedded features being derived by an encoder from the positive and negative samples. The method according to claim 8 , wherein the contrast weight map is based on the similarity of each location in the ultrasound image with a positive sample and the dissimilarity with a negative sample.

12. The method according to claim 1, wherein the information comprises a relationship graph formed by positive samples and negative samples.

13. A medical system for interactive segmentation, the medical system comprising: a memory configured to store the medical image; User Input, configured to receive changes from the user; an image processor configured to receive, from a user input, a change from a first position to a second position for a first segmentation of the medical image, and use the first patch for the first position and the second patch for the second position for a second segmentation of the medical image, the second segmentation formed by the machine learning network; as well as The display is configured to display the second segmented segmented image.

14. The medical system of claim 13, wherein the medical image comprises a magnetic resonance image, an ultrasound image, a computed tomography image, or an x-ray image. 15 . The medical system of claim 13 , wherein the alteration is a change in the positioning of a boundary of the first segmentation, wherein the first patch is at the location of the boundary before the alteration and the second patch is at the location of the boundary after the alteration.

16. The medical system of claim 13, wherein the machine learning network comprises an image-to-image network having a first input for the medical image and a second input for a relationship graph that relates each location in the medical image and a similarity to the first patch and the second patch.

17. The medical system of claim 16 , wherein the image processor is configured to generate feature embeddings for a first patch, a second patch, and patches from different locations in a medical image, the relationship graph being formed by weights that maximize similarity of the feature embedding of the patch at the different location to the feature embedding of the second patch, the similarity being weighted by similarity of the feature embedding of the patch at the different location to the feature embedding of the first patch.

18. A method for artificial intelligence-assisted contouring in a medical imaging system, the method comprising: generating a graph representing: (a) a similarity of each location in the medical image to a first location of a first contour of the object in the medical image after the alteration to the first contour and (b) a dissimilarity of each location to a second location of the first contour of the object in the medical image before the alteration; segmenting the object by the machine learning model in response to input of the medical image and the graph to the machine learning model; as well as Displays the segmented object.

19. The method of claim 18, wherein generating the map comprises generating the map using embedded features of the patch for each position, the first location, and the second location.

20. A method according to claim 18, wherein the change to the first contour is made by a user to a portion of the first contour, wherein other portions of the first contour are not changed by the user, and wherein the segmentation includes segmenting the portion and the other portions without requiring user input relative to the other portions.

Images