Artificial intelligence support contouring in medical imaging

Abstract

To provide a method of supporting segmentation in a medical imaging system.SOLUTION: A method according to the present invention has steps of, if a user changes segmentation of an object, using an index indicating what is not desired (e.g., a sample from a place which does not exist any more due to the change but previously had a segmentation boundary) in order to provide a notification of a segmentation relating to an unchanged region, extracting negative and positive samples from a user change, and using the information based on the negative sample regardless of whether there is any positive sample by a machine learning model for re-segmentation of the object. The user change is used for modification of segmentation even in a region without any change. It is possible to provide segmentation unique to the user and / or the object while minimizing a required manual adjustment amount.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] The embodiments presented herein relate to segmentation in medical imaging. Contouring structures is a fundamental task in medical imaging. This segmentation allows a user to focus on regions of interest and / or measure various attributes of the structures. Segmentation aids in diagnosis and treatment planning.

[0002] Clinicians typically manually contour arbitrarily selected structures. Structure contouring is task- and user-dependent. For example, in the case of left atrial appendage (LAA) contouring in intracardiac echocardiography (ICE), an ultrasound technician may tightly contour the LAA to a clear image boundary. In an interventional setting, for safety purposes, physicians tend to broaden the structure contour to include more surrounding tissues to account for potential structural deformation over time (cardiac cycle, time from device fabrication to implantation). A given segmentation model or process cannot perform separate segmentations for different purposes. Therefore, users are forced to modify the segmentation manually in a time-consuming, user-dependent manner.

[0003] In the field of interactive segmentation, the user adjusts the contours. Artificial intelligence (AI) has shown promising performance for image segmentation. AI can assist in interactive segmentation. User input is converted into image maps, such as Gaussian heat maps, Euclidean distance maps, geodesic distance maps, or simply small surface plates. These maps are input along with the initial image to generate a segmentation based on the user changes. If the user makes further edits for correction purposes, a similar map is generated and used to generate an updated segmentation. These approaches focus only on the corrections and lose information about why the user made the corrections. Summary of the Invention

[0004] A system, method, and non-transitory computer-readable medium are provided for AI-assisted segmentation. When a user modifies an object's segmentation, indicators of what is undesirable (e.g., samples from where a segmentation boundary previously existed but no longer exists due to the modification) are used to inform the segmentation for areas without modification. Negative and positive samples are extracted from the user modification. Information based on the negative samples, with or without positive samples, is used by a machine learning model to re-segment the object. The user modification is used to revise the segmentation even for areas without modification, minimizing the amount of manual adjustment required and providing a segmentation specific to the user and / or purpose.

[0005] In a first aspect, a method for assisted segmentation in a medical imaging system is provided. An ultrasound image of a patient is acquired. An object in the ultrasound image is segmented, resulting in a first contour of the object. A modification of the first contour of the object is received from a user input. Positive and negative samples are extracted from the ultrasound image based on the modified contour. The object in the ultrasound image is segmented by inputting the ultrasound image and information about the positive and negative samples into a first machine learning model. The segmentation results in a second contour. An image of the second contour is generated.

[0006] In a second aspect, a medical system for interactive segmentation is provided. A memory is configured to store a medical image. A user input is configured to receive a change from a user. An image processor is configured to receive from the user input a change from a first location to a second location for a first segmentation of the medical image, and to use the first patch for the first location and the second patch for the second location for a second segmentation of the medical image. The second segmentation is generated by a machine learning network. A display is configured to display a segmentation image of the second segmentation.

[0007] In a third aspect, a method for AI-assisted contouring in a medical imaging system is provided. A map is generated representing (a) a similarity of a location in a medical image to a first position of a first contour of an object in the medical image after modification of the first contour, and (b) a dissimilarity of the location to a second position of the first contour of the object in the medical image before modification. The object is segmented by a machine learning model in response to inputting the medical image and the map to the machine learning model. The segmented object is displayed.

[0008] The exemplary embodiments listed below incorporate other features or aspects. One or more aspects described above or in the exemplary embodiments may be used alone or in combination with other exemplary embodiments, features, or aspects. Any aspect or feature of a method, system, or computer-readable medium may be used in other methods, systems, or computer-readable mediums. These and other aspects, features, and advantages will become apparent from the following detailed description of the preferred embodiments, which is described in conjunction with the accompanying drawings. The present invention is defined by the claims, and nothing in this section should be construed as limiting the scope of the claims. Additional aspects and advantages of the present invention will be described below in connection with the preferred embodiments and may be later claimed alone or in combination. [Brief explanation of the drawings]

[0009] The components and drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the embodiments. Moreover, in the drawings, like reference characters indicate corresponding parts throughout the various views. [Figure 1] 1 is a flowchart of one embodiment of a method for AI-assisted contouring in medical images. [Figure 2] 10 is a flowchart of another embodiment of a method for AI-assisted contouring. [Figure 3] 1 is a flowchart of one embodiment of using user-modified positive and negative samples for AI segmentation. [Figure 4] 10 illustrates user modifications to a contour. [Figure 5] Illustrate segmentation progression based on manual and AI-assisted changes. [Figure 6] FIG. 1 is a block diagram of one embodiment of a medical system for AI-assisted interactive contouring. Detailed Description of the Embodiments

[0010] Given an initial contour (e.g., generated by an AI), the user only needs to locally adjust the contour. Negative samples of the adjustment, with or without positive samples, reflect the user's intention for the rest of the contour. The samples are used by the AI ​​or another AI to understand the user's intention and adjust the entire contour globally.

[0011] In the example used here, interactive contouring is provided for three-dimensional (3D) or four-dimensional (4D) ultrasound of anatomical structures. Starting with a single user click, the system first seeks to generate an initial contour of the target using a spatio-temporal AI segmentation model. If the user disagrees, the user refines local segments of this initial contour. This refinement model generates positive and negative samples (image patches around the modified segments) from the initial contour and manually modified segments of the contour. A matching approach for the positive and negative samples is used by a deep network model to efficiently screen and update the remainder of the contour (e.g., segmentation is performed using the image and information from the modified samples).

[0012] Rather than focusing only on encoding segments when making modifications (i.e., positive samples), we use negative samples, or both negative and positive samples, to learn about the user's intent and globally update the contour to match this intent. The user edits (positive samples) and the reasons for the edits are also used by looking at the negative samples. Using a contrastive weight map based on positive and negative samples, the proposed approach generates high attention (e.g., high weight scores in the contrastive weight map) for patches that have high similarity to the user's intent for the contour (positives) and low attention for patches where the user makes modifications (negatives).

[0013] Rather than inputting the edited contours and the original image through a segmentation network, relational map encoding is used. Relational map encoding is based on positive and negative examples and challenges the machine learning network to learn the user's intent. The network does not know which parts of the image to focus on. Machine training with contrastive weight maps gives the network clear instructions in the intent: which image patches the network should direct its attention to and which ones it should not. The resulting machine learning model is trained to use the positive and negative samples for each local change in segmentation to inform the global segmentation of the object.

[0014] Figure 1 is a flowchart of one embodiment of a method for assisted segmentation in a medical imaging system. The method uses AI-assisted contouring. Negative samples of segmentation changes, with or without positive samples, are extracted to reflect the user's intent regarding where the contour should not be or where the contour should be. The AI ​​uses the samples and the medical image to re-segment based on the user's intent.

[0015] The method of Figure 1 is performed by a medical system, such as the medical scanner or image processor of Figure 6, or other medical system. For example, an ultrasound scanner acquires an image. An image processor segments, receives input regarding modifications from a user input, extracts samples from the modifications, and generates an image of the updated segmentation. The image processor may generate the image. A display displays the image based on the image generated by the image processor.

[0016] Although the method is performed in the order shown (e.g., top to bottom or numerical order), other orders may be used. Additional, different, or fewer steps may be provided. For example, step 120 is not provided if the acquired image includes segmentation, such as was previously performed. In another example, step 152 is not used, such as if samples are input directly without mapping. In yet another example, step 160 is not performed, such as if the refined segmentation is saved in the patient medical record for later review. In yet another example, steps are performed to configure the scanning and / or use of the output segmentation information.

[0017] FIG. 2 is a flowchart of one implementation of the method of FIG. 1 in one context or workflow. FIGS. 1 and 2 are described together. Other contexts or workflows may also be used with the method of FIG. 1. In FIG. 2, the workflow involves interactive contouring of ultrasound images. Starting with a two-dimensional (2D) image, such as a 2D multiplanar reconstruction (MPR) from a 4D ultrasound sequence, an AI module generates an initial contour of the target in step 120. An editing loop then begins with an interactive update in step 200. The user edits the contour segments to include more or less of the adjacent soft tissue in step 130. A sample generation module generates positive and negative samples (e.g., image patches) around the edited region in step 140. The AI ​​module then uses these samples as input to generate an updated overall contour result in step 150 based on ray-tracking sampling of image space in step 210. If the user makes further edits, the loop continues in step 220 until satisfied.

[0018] In process 100, a medical scanner (e.g., an ultrasound scanner or imaging system) or image processor acquires a medical image representing a patient. The patient is scanned. The resulting image is a medical image. In other embodiments, the medical image of the patient is retrieved from memory or transferred over a computer network.

[0019] The medical image may represent any part of a patient, such as a representation of the patient's organs, head, or torso. In one embodiment, the medical image is an ICE image acquired using an ICE transducer (e.g., a transducer in an ICE catheter scans from within the patient's heart). The ultrasound image represents a region of the patient's heart, such as imaging the patient's ostium, heart chambers, and / or valves, and / or devices within the patient. ICE provides real-time, high-resolution visualization of cardiac structures and continuous monitoring of catheter positioning within the heart. Procedural complications, such as pericardial effusion and thrombus formation, may be detected early using ICE. ICE exhibits excellent patient tolerance, reduces fluoroscopy time, and eliminates the need for general anesthesia or a second operator. Other ultrasound imaging techniques, such as transesophageal echocardiography (TEE), may also be used. In other embodiments, the medical image is a magnetic resonance image, ultrasound image, computed tomography image, or x-ray image.

[0020] The medical images and corresponding segmentations may have any spatial extent, such as 2D or 3D. Temporal variations may also be provided, such as segmenting through a series of images over one or more cardiac cycles. A 2D example is used here. A static 2D image example is used here.

[0021] A medical image is data arranged as pixels or voxels, which may or may not be formatted for display. A medical image can be data acquired by scanning, with or without further processing (e.g., scan conversion). A medical image is data used to generate a display image that represents a patient, or is a display image.

[0022] In step 120, the image processor segments objects in the ultrasound image. This segmentation identifies a boundary, area, and / or volume corresponding to the object. After segmentation into a boundary, area, and / or volume, the object has a contour or outer (perimeter) edge (curve or surface). The segmentation identifies pixels, voxels, or scan locations corresponding to the object. The result of the segmentation is a contour that, together with other objects, represents the outer edge or boundary of the object.

[0023] One or more objects may be segmented. In the ICE example, an ostium, a valve, or a chamber may be segmented. Soft tissue structures (e.g., organs or muscles), bones, devices, or implants may be objects represented in a medical image and segmented.

[0024] Segmentation can be performed using any process or function, such as intensity thresholding with low-pass filtering. In one embodiment, full width at half maximum (FWHM) or intensity thresholding with various standard deviations is used for segmentation. Alternatively, a user (e.g., a radiologist) or an image processor performs the segmentation. In another approach, a machine learning model, such as an encoder-decoder-based neural network, outputs one or more segmentations in response to a medical image input. Image-to-image, U-Net, or encoder-decoder networks trained to output segmentations in response to medical image data input can be applied. A machine-trained model (segmenter) generates values ​​for features in a hidden layer in response to a medical image input and uses the feature values ​​to output a segmentation.

[0025] The segmentation outputs an identification of the locations occupied by the object. The contour (e.g., boundary, area distribution, or volume distribution) of the object is output as the segmentation. The output contour is based on or generated in response to an input or displayed medical image.

[0026] The segmentation is displayed to the user. The user may modify the segmentation based on the user's desires and / or goals. For example, the user visually identifies one or more locations where the contour covers the object too broadly or insufficiently. An example is shown in FIG. 4. The initial segmentation 400 is roughly elliptical, but includes a notch or depression formed by the contour 420 in portion 430. In another example, the segmentation of the LAA is not tight to the boundary. Because the physician is performing measurements or diagnosis, the contour should be tight to the LAA boundary. In yet another example, the segmentation of the LAA is tight to the boundary. Because the physician is planning or performing an interventional procedure, a more comprehensive contour is desirable to account for potential structural deformations.

[0027] In step 130, the image processor receives the segmentation modifications. The contours are modified, for example, only part of the contours are modified.

[0028] The modifications can be automated, such as by using another machine learning model to identify the modifications. In one approach to user interaction, the modifications are received from the user via a user input. The user modifies the contour using click-and-drag or other user interface manipulations with a user input (e.g., a mouse, trackball, touchpad, touchscreen, and / or keyboard). In the example of FIG. 4, the user selects a point or location 410A along the contour 420. The user then drags the point or location 410A to the desired point or location 410B. Only portion 430 of the contour 420 is modified to segment it based on this modification. The contour can have elasticity defined using a physics model or by another function that relates the modification of a point or region to adjacent portions of the contour 420 (i.e., within portion 430). Other portions of the contour 420, such as outside portion 430, remain unchanged.

[0029] The user selects location 410A as a representative modification. Rather than modifying the entire desired location, one or more representative modifications are made. In this representative region, one or more locations of points are shifted, such as inward or outward relative to the object. The remainder of the contour remains unchanged by the user input. An example is shown in FIG. 5. An initial contour 500 based on the segmentation of step 120 includes two depressions 502 and 504. The user selects a point within depression 504 to modify the contour. In this modified contour 510, notch 504 is gone, but notch 502 remains. Although shown as notches, a combination of notches and / or protrusions may occur.

[0030] The changes (or modifications) indicate the user's intention regarding the current segmentation. In step 140, the image processor extracts positive and / or negative samples from the image based on the modified contours. The image processor uses information from the medical image based on the changes to indicate the user's intention regarding the segmentation.

[0031] Samples are extracted from a medical image, such as an ultrasound image. The samples are patches of any size, such as 4x8, 8x8, or patches of other sizes or shapes. Other samples, such as sparse sampling, may also be obtained. The samples may be directional, such as having a longer range parallel or perpendicular to the contour.

[0032] Any number of samples may be extracted. For example, one or more positive samples and one or more negative samples may be extracted. The resolution of the medical image and / or the size of the portion 430 may control or influence the number of samples. For example, a sample may be extracted at each point of the image resolution along the contour of the modified portion 430. A set number of samples may also be obtained, such as by dividing the modified portion into a set number of parts and then sampling along the contour.

[0033] Positive samples may be extracted. For the portion 430 of the contour 400 that is modified, samples along the modified contour are extracted. The user has modified the segmentation so that the contour follows the desired structure. The samples extracted from the modified contour 420 reflect the positive placement of the contour 400.

[0034] Negative samples are extracted. For the portion 430 of the modified contour 400, samples are extracted along the original contour 420. The user has modified the segmentation so that the contour 400 does not follow undesired tissue. The samples extracted from the contour 400 for the original portion 430 reflect the negative or undesired placement of the contour 400. Other undesired or negative samples may be extracted, such as tissue that is centered away from the contour (e.g., inward or outward from the contour relative to the object and not along the modified contour).

[0035] In one embodiment, there are two segments of the contour: the edited segment and the remaining portion. For the edited segment, image patches centered around the contour to be modified are extracted. These are positive samples because the user intends to move the contour to these kinds of patches. Image patches centered around the original corresponding contour segment before editing are extracted. These are negative samples because the user intends not to have the contour pass through these patches.

[0036] The extracted samples represent samples that indicate contour and non-contour tissue locations. These samples can be used to assist in identifying contour locations in other parts of the contour. The ultrasound image can be sampled in any manner to test segment locations. In one approach, shown in FIG. 2, radial sampling is used. The center of the object to be initially segmented is used as the center of the radial lines. The contour locations along each radial line are found by sampling along the radial line. The extracted samples can be used to assist in identifying locations along the radial line corresponding to the radius. For remaining segments that the user does not edit, radial patches centered around the contour can be extracted. The positive and negative samples are used to identify, for each ray or radial line, which patches along the ray should be suitable patches for placing the contour on. Other sampling of the medical image can be, for example, along a Cartesian coordinate grid.

[0037] In step 150, an image processor segments an object in a medical (e.g., ultrasound) image. The segmentation results in another contour for the same object that is likely, but not necessarily, different from the previous contour based on the segmentation in step 120. This subsequent segmentation in step 150 uses information from the medical image and the sample as input to generate a contour. This contour provides a boundary, area, volume, or other designation of location for the object that is distinct from other locations.

[0038] The segmentation pertains to the portions not modified by the user. The contour or the entire object may be segmented to provide contours for the portions that are modified and other portions. Information from samples of the entire medical image is used to identify the contour (segmentation) for the entire object or other portions, without specific user input for these other portions. Figure 5 shows an example in which an initial contour 500 is modified by the user to become contour 510. In step 150, the image processor re-segments the object using the samples and the medical image based on the user modifications, resulting in contour 520. Contour 520 does not include dip 502 and does not include dip 504.

[0039] The same object will be segmented again after the modifications of step 130. Samples based on the modifications are used for this re-segmentation. The samples are either directly input as information into the segmentation process or are used to derive further information. This further information is directly input into the segmentation process. Step 152 shows one way in which further information is derived (generating a map) for input into the segmentation process.

[0040] Any of the segmentation processes or approaches described above for process 120 can be used in the segmentation of process 150. The same or a different approach can be used in process 150 than in process 120. In one embodiment, the segmentation of process 150 uses a machine learning model that uses input related to the medical image and information from the sample. The segmentation is generated by the machine learning model in response to input from two or more input channels, one for the medical image and one for information from the sample related to changes made by a user.

[0041] Machine learning models are pre-trained using training data, which are formed from an architecture that defines learnable parameters. The training data includes many examples of inputs and corresponding ground truth (e.g., correct outputs), such as expert-annotated segmentations generated from a database of patient scans.

[0042] To train a machine learning model, a machine learning model arrangement (architecture) is defined. Any currently known or later developed machine learning model may be used. For example, an image-to-image network, U-Net, DenseNet, ResNet, or an encoder-decoder network may be used. Downsampling layers, convolutional layers, pooling layers, dropout layers, skip connections, upsampling layers, and / or other neural network layers may be used. Any architecture that receives image (spatial) information to output spatial (e.g., image) information may be used. The definition is based on the configuration or programming of the learning. The number of layers or units, the type of learning, and other characteristics of the model are controlled by a programmer or user. In other embodiments, one or more schemes (e.g., the number of nodes, the number of layers or units, or the type of learning) are machine-defined and selected during learning. Training data containing many samples of input data and corresponding outputs is used for training. The relationship between the input and output is machine-learned.

[0043] An image processor or another processor machine trains the model. Training involves learning values ​​for learnable parameters (e.g., weights, connections, filter kernels, and / or other learnable parameters of a defined architecture). Deep or other machine learning may be used. The weights, connections, filter kernels, and / or other parameters are features to be learned. For example, convolution kernels are features to be trained. Using training data, values ​​of the learnable parameters of the model are adjusted and tested to determine the values ​​that lead to the best estimate of the output given the input. Adam or another optimization method is used for training.

[0044] During training, a loss is minimized. The loss is derived from comparing the model's output to the expected output (i.e., ground truth). The loss function is an L1, L2, or other error function between the network output and the ground truth in the training data. Other losses can be used. Using optimization, multiple values ​​of the learnable parameters are tested across multiple samples of the training data to minimize the loss (or maximize the reward).

[0045] In one embodiment, the machine learning model includes input information from samples, including negative samples or samples of the contour before the user's modification but not after the user's modification. This input can be the samples themselves or information derived from the samples. For example, a map is derived from the samples in step 152. The map indicates the user's intent regarding the location of the modification and / or other locations. For example, the map compares the extracted samples to the medical image at sampled locations throughout the medical image. Sampling can be limited to within a threshold distance (e.g., 10% or 10 mm) of the initial contour from step 120, or the entire image can be fully or sparsely sampled. The results of the comparison, such as similarity, are used to derive a contrast value or weight for each location. In an example of radial line sampling, various radial patches are sampled along each of various radial lines. Each position along each of the radial lines is sampled. Sampling can be limited to those within a threshold distance from the previous contour. A patch centered at each sample location is extracted or defined.

[0046] The radial or sample patch is compared to the negative samples. A large similarity or small dissimilarity indicates that the contour should not be placed there. Comparison with positive samples can also be used. A large similarity or small dissimilarity of the radial patch to the positive samples indicates that the contour should be placed there. If many positive or negative samples are used, an average score can be used. If both positive and negative samples are used, a combination of the comparison results can be used. Some combination, such as a weighted difference or ratio, can be used. A map is generated in step 152 as a weight or relationship value by sample location of the medical image to the positive and / or negative samples from the alteration location. Each location along or around the object is sampled and compared to generate a map of the spatial distribution of similarity (dissimilarity) to the extracted negative and / or positive samples. The map can then be used as input to a machine learning model.

[0047] In one approach to step 152, the image processor generates a map representing (a) the similarity of locations in the medical image to the position of the object's contour in the medical image after the contour has been modified (positive samples), and (b) the dissimilarity of locations to the position of the object's contour in the medical image before the contour has been modified (negative samples). User-initiated contour modifications to one portion of the contour are used to map other portions of the contour that remain unchanged by the user. A relational map is formed using the positive and / or negative samples. When both positive and negative samples are used, the contrastive weight map is based on the similarity of image locations to the positive samples and the dissimilarity to the negative samples. A combination of similarity and dissimilarity measures can be used. Any measure of similarity or dissimilarity can be used, such as the minimum sum of absolute differences or a cross-correlation coefficient. The contrastive weight map is formed from the positive and negative samples, and a machine learning model outputs contours in response to inputs of the image and the contrastive weight map.

[0048] 3 shows an example of using samples to generate a map in process 152. The samples' embedding features 332, 334, and 336 are used in comparison to form a contrast weight map 340. The edited segment or contour 300 is used in process 140 to extract positive samples 322 as patches around the edited contour portion. The unedited contour portion is used in process 140 to extract negative samples 324 as patches around the unedited contour. In process 320, patches 326 are extracted from the remainder of the segmentation, such as within a threshold distance from the previous segmentation along a radial line, and / or from a sampling of the medical image that includes locations away from the portion to be changed by the user.

[0049] The map uses embedding features 332, 334, and 336 from patches 322, 324, and 326. The embedding features 332 (patch 322) from the positive samples, the embedding features 334 (patch 324) from the negative samples, and the embedding features 336 from sample patch 326 are used to form map 340. The embedding features 332, 334, and 336 are generated or output by encoder 330 or another neural network. The image processor generates latent representations from the input patches 322, 324, and 326. The patches 322, 324, and 326 are input to encoder 330. In response to the input, encoder 330 generates latent representations 332, 334, and 336 as fingerprints.

[0050] The encoder 330 is a convolutional neural network and a transformer. In other approaches, other encoders formed by other neural networks, such as a fully connected neural network with a softmax output layer or a multi-layer perceptron or a convolutional neural network, can be used. A deep learning neural network can be used for the encoder 330.

[0051] The encoder 330 operates based on how it has been trained. Different training results in different values ​​for the learnable parameters and therefore different operation. In one approach, the encoder 330 or another machine learning model that generates the latent representations for the patches 322, 324, and 326 is trained with the decoder (e.g., the encoder 330 is trained as part of a segmentation network). Alternatively, the encoder 330 is trained using many examples of patches as training data, where the ground truth is binary (0 or 1), positive or negative. The encoder 330 is trained to distinguish between positive and negative input samples.

[0052] After training, features 332, 334, and 336 are extracted from layers of the encoder 330 before the output layer (e.g., before a softmax or multilayer perceptron or before the decoder). For example, features from one or more layers after the input layer and before the output (i.e., binary 1 or 0) layer of the encoder 330 are used. Alternatively, the encoder 330 outputs latent representations, such as when trained as part of an encoder-decoder. The latent representations formed by the encoder 330 are used as feature vectors 332, 334, and 336. Features from one or more hidden layers of the encoder 330 are used. The same encoder 330 with the same learning parameter values ​​is used for each patch of each type. In an alternative approach, different encoders 330 are trained for different types of patches (e.g., positive, negative, and image samples).

[0053] Given positive and negative samples (patches 322 and 324) and sample (e.g., radial) patches 326 from an initial or previous segmentation, these patches 322, 324, 326 are input to a feature encoder 330 with shared weights for different types of patches 322, 324, 326. Deep feature embeddings (feature vectors 332, 334, 446) are obtained for each patch.

[0054] A contrastive weight map 340 is constructed from these features in step 152. For each sample location with a corresponding patch 326, a score is generated from the features of that patch 326, the features of the positive patch 322, and the features of the negative patch 324. The score increases the similarity between the sample patch 326 and the positive sample patch 322 and decreases its similarity to the negative sample patch 324, where the similarity is based on the embedding features. This map 340 then helps a downstream machine learning model 360 (network) to increase attention to regions with higher scores and decrease attention to regions with lower scores (or vice versa, depending on how the map is generated). The weight map 340 is concatenated with the original image 350, and together they are input into a segmentation network (machine learning model for segmentation 360), which outputs an updated segmentation 370.

[0055] In one embodiment, the contrast weight is a score that maximizes the similarity of the sample (e.g., radial) patch embedding 336 to the positive patch embedding 332 while minimizing the similarity of the sample (e.g., radial) patch embedding 336 to the negative patch embedding 334. m , negative samples n1,n2,…,n s , and radial (sample) patches r1,r2,…,r T Given, these image patches are input to a feature encoder 330, such as a convolutional neural network followed by a transform encoder. The superscript,f, denotes the feature embedding 332, 334, 336, i.e., f r i ,f p j ,f n k The contrastive weight of the i-th radial patch 326 is defined as: JPEG2025157135000002.jpg32117 where τ is a temperature parameter. The numerator indicates the similarity of the radial patch (after encoding) to the positive samples, and the denominator indicates the dissimilarity of the radial patch to the negative samples. A weighting factor or score is determined for each radial patch 326. Other functions can also be used. The weights in the contrastive weight map 340 are the spatial distribution of the sample patches 326 that direct or inform the downstream segmentation network 360 to pay appropriate attention to relevant parts of the image. The weights distinguish possible positive locations of the contour from possible negative locations based on the changes made by the user. The weight map is formed by calculating scores for all sample (e.g., radial) patches 326. For each different location sample from the image, a weight is generated from maximizing the similarity of the location sample to the positive samples, inversely weighted by the similarity of the location sample to the negative samples.

[0056] The contrastive weight map 340 and image 350 are input into a machine learning model (segmentation network 360), which then outputs a contour. The segmentation is updated globally based on local changes made by the user. The machine learning model 360 is trained using many samples of information from the input image and extracted patches with the ground truth segmentation. If the user desires further edits to the contour, as shown in Figure 2, the process is repeated until the user's final segmentation target is achieved.

[0057] In step 160 of Figure 1, an image processor generates an image. The image shows a segmentation (outline). The segmentation can be a graphic, such as a boundary or shape of an object. The segmentation can also be a modification of the medical image, such as coloring or highlighting the segmented object. In one approach, a graphic of the segmentation is overlaid on the medical image. In another approach, a representation of the object is extracted from the image based on the segmentation. A representation of only the object is displayed as the segmentation.

[0058] The segmented object is displayed to the user, who may use the segmentation for measurement, diagnosis, treatment planning, or other purposes.

[0059] 6 shows an example of a medical system for interactive segmentation. The medical system includes a display 630, a user input 620, a memory 610, and an image processor 600. The display 630, the image processor 600, the user input 620, and / or the memory 610 may be part of a medical scanner 640 (e.g., an ultrasound scanner), a computer, a server, a workstation, or other system that processes medical images from a patient scan. A workstation or computer without a medical scanner 640 may also be used as the medical system.

[0060] Additional or different components may be provided, or fewer components may be included. For example, a computer network may be included for remote image generation of locally captured image data or for local prediction from remotely acquired image data. The machine learning model 605 may be applied as a standalone application on a workstation or local device, or as a service deployed in a network (cloud) architecture. In another example, the medical scanner 640 is not provided.

[0061] Images (e.g., ultrasound images), segmentations, user edits, learning parameter values, machine learning model 605, feature values ​​(e.g., latent expressions), positive samples, negative samples, image samples, display images, and / or other information are stored in a non-transitory computer-readable memory such as memory 610. Medical images (e.g., magnetic resonance images, ultrasound images, computed tomography images, or X-ray images) are stored in memory 610. Memory 610 may be external storage, RAM, ROM, a database, and / or local memory (e.g., a solid-state drive or hard drive). One or separate non-transitory computer-readable media may be used for instructions and other data. Memory 610 may be implemented using a database management system (DBMS) and may reside in memory such as a hard disk, RAM, removable media, etc. Alternatively, memory 610 may be internal to processor 600 (e.g., cache).

[0062] The instructions for implementing the training or application process, methods, and / or techniques described herein may be provided in a non-transitory computer-readable storage medium or memory, such as a cache, buffer, RAM, removable media, a hard drive, or other computer-readable storage medium (e.g., memory 610). Computer-readable storage media include various types of volatile and non-volatile storage media. The functions, acts, or tasks illustrated in the figures or described herein are performed in response to one or more instruction sets stored in a computer-readable storage medium. The functions, acts, or tasks are not dependent on a particular type of instruction set, storage medium, processor, or processing strategy, and may be performed by software, hardware, integrated circuits, firmware, microcode, etc., operating alone or in combination.

[0063] In one embodiment, the instructions are stored on a removable media device for reading by a local or remote system. In another embodiment, the instructions are stored at a remote location for transfer over a computer network. In yet another embodiment, the instructions are stored within a given computer, CPU, GPU, or system. Some of the components and method steps making up the systems shown in the figures may be implemented in software, so the actual connections between system components (or process steps) may vary depending on the manner in which the embodiment is programmed.

[0064] The user input 620 includes a keyboard, buttons, sliders, dials, trackball, mouse, touchpad, touchscreen, and / or another device for receiving user input in a user interface. The user input may be configured by the image processor 600 or another processor and may receive modifications from a user. A segmentation is displayed. The user interface may be configured to allow a user to modify the segmentation, such as by modifying a portion of a contour. The position of the boundary of the segmentation is modified by the user based on the received user input, such as by click-and-drag interaction with the displayed contour. More than one modification may also be made.

[0065] Image processor 600 may be a controller, control processor, general-purpose processor, microprocessor, tensor processor, digital signal processor, three-dimensional data processor, graphics processing unit, application specific integrated circuit, field programmable gate array, artificial intelligence processor, digital circuit, analog circuit, combination thereof, or other now known or later developed device for processing image data. Image processor 600 may be a single device, multiple devices, or a network of devices. In the case of two or more devices, parallel or serial division of processing may be used. The multiple devices comprising image processor 600 may perform different functions. In one embodiment, image processor 600 is a control processor or other processor of medical scanner 640. Image processor 600 operates according to and is configured by stored instructions, hardware, and / or firmware to perform the various processes described herein.

[0066] The image processor 600 or another remote processor is configured to train the machine learning architecture. Based on the network architecture and user-provided or other sources of training data (e.g., generated from an expert-curated database of patient scans and ground truth), the image processor 600 learns to associate one or more input variables (e.g., images or information from images and negative samples) with an output (e.g., a segmentation or distinguishing positive samples from negative samples). The result of training is a machine learning model 605, e.g., a model for generating a segmentation in response to inputs of negative samples or information derived therefrom and medical images. Others of the machine learning models described herein (e.g., the encoder 330 of the initial segmentation or a segmentation network that does not use information from user-modified samples) may be trained by the image processor 600 and / or stored in memory 610.

[0067] Alternatively or additionally, the image processor 600 is configured to apply a machine learning model 605 (i.e., apply a segmentation model with input of medical images, apply an encoder to generate embedding features for patches, and / or apply a segmentation model with input of information derived from positive and / or negative samples based on medical information and user modifications). The machine learning model 605 may include a machine learning network for global segmentation based on local user modifications.

[0068] The image processor 600 is configured to receive a change from one location to another location for a segmentation of the medical image. The change is received from a user input 620. The image processor 600 is configured to use the patch for the one location and the patch for the other location for another segmentation of the medical image. The other segmentation is generated by a machine learning network or model 605. The patch for the one location is at the location of the boundary before the user's change, and the patch for the other location is at the location of the boundary after the user's change.

[0069] In one approach, the machine learning network or model 605 is an image-to-image network (e.g., U-Net or encoder-decoder) that has one input of a medical image and another input of a relational map that associates positions in the medical image with similarity to patches from one location (negative) and patches from another location (positive).

[0070] In a further approach, the image processor is configured to generate feature embeddings for patches (patches for one location, patches for another location, and patches sampled from different positions within the medical image). The relational map is formed from weights that maximize the similarity of the feature embedding of the patch at each sample position to the feature embedding of the patch based on the contour after modification, weighted (e.g., divided) by the similarity of the feature embedding of the patch at each sample position to the feature embedding of the patch based on the contour before modification by the user.

[0071] The image processor 600 is configured to generate an output such as an image showing the segmentation. The contours of the objects are displayed alone or together with other information such as an overlay on the medical image or an image of the object extracted from the medical image based on the segmentation. The image processor 600 is configured to generate a visual display of the segmentation (contours).

[0072] The display 630 may be a CRT, LCD, projector, plasma, printer, tablet, smartphone, or other now known or later developed display device for displaying output such as images of the segmentation (segmentation images). Images of the previous segmentation, modified, and / or user modified segmentation may be displayed sequentially and / or simultaneously.

[0073] The medical scanner 640 is a diagnostic or therapeutic scanner (e.g., ultrasound, computed tomography, x-ray, or magnetic resonance scanner). The scanner 640 operates according to one or more settings to scan a patient 642 resting on a bed or table 644. The settings control the scan, including transmission, reception, reconstruction, and image processing. A scanning protocol is followed to generate data representative of the patient 642, such as an ICE ultrasound image representing the patient's cardiac region. The patient 642 is imaged by the scanner 640 using the settings. The scanner 640 generates medical images representing the object to be segmented.

[0074] The following is a non-limiting list of exemplary embodiments disclosed herein. Example embodiments of one type of set (e.g., method or system) may also be provided in or combined with other types of sets of exemplary embodiments.

[0075] Example embodiment 1: 1. A method for assisted segmentation in a medical imaging system, comprising: obtaining an ultrasound image of the patient; segmenting an object in the ultrasound image, the segmentation resulting in a first contour for the object; receiving a change to a portion of the first contour for the object from a user input; extracting positive and negative samples from the ultrasound image based on the modified contour; segmenting the object in the ultrasound image, the segmentation resulting in a second contour by inputting information about the ultrasound image and the positive and negative samples into a first machine learning model; generating an image of the second contour.

[0076] Example embodiment 2: The method of Example Embodiment 1, acquiring the ultrasound image representing a cardiac region of the patient using an intracardiac echocardiographic transducer; The method, wherein the object comprises an ostium, a heart chamber, or a valve.

[0077] Example embodiment 3: The method of any one of Example Embodiments 1 to 2, The method, wherein the segmentation resulting in the first contour includes segmenting by a second machine learning model that outputs the first contour in response to input of the ultrasound image.

[0078] Example embodiment 4: The method of any one of exemplary embodiments 1 to 3, receiving includes receiving a user input change that shifts a location of a point on the first contour away from the first contour; The method, wherein the portion includes the location and adjacent locations that are shifted with the modification, and other portions of the first contour are unchanged.

[0079] Example embodiment 5: The method of any one of Example Embodiments 1 to 4, The extracting step comprises: extracting the positive samples as one or more patches in the ultrasound image that include the portion of the first contour after the modification; extracting the negative samples as one or more patches in the ultrasound image that are separate from the first contour after the modification.

[0080] Example embodiment 6: The method of any one of Example Embodiments 1 to 5, The method, wherein the extracting includes extracting the negative samples as one or more patches in the ultrasound image that include the portion of the first contour before the modification.

[0081] Example embodiment 7: The method of any one of Example Embodiments 1 to 6, The method, wherein the segmentation resulting in the second contour comprises segmentation by the first machine learning model including a U-Net.

[0082] Example embodiment 8: The method of any one of Example Embodiments 1 to 7, The segmentation resulting in the second contour comprises: forming a contrast weight map from the positive and negative samples; outputting the second contour by the first machine learning model responsive to input of information from the ultrasound image and the contrastive weight map.

[0083] Example embodiment 9: 10. The method of example embodiment 8, further comprising: The method, wherein forming includes generating a weight for each different location sample from the ultrasound image from maximizing the similarity of the location sample to the positive samples, inversely weighted by the similarity of the location sample to the negative samples.

[0084] Example embodiment 10: The method of any one of exemplary embodiments 8-9, The method, wherein the forming includes forming the contrastive weight map from embedded features derived from the positive samples and negative samples by an encoder.

[0085] Example embodiment 11: The method of any one of exemplary embodiments 8 to 10, The method, wherein the contrastive weight map is based on similarity of locations in the ultrasound image for the positive samples and dissimilarity for the negative samples.

[0086] Example embodiment 12: The method of any one of Example Embodiments 1 to 11, The method, wherein the information includes a relational map formed using the positive and negative samples.

[0087] Example embodiment 13: 1. An interactive segmentation medical system, comprising: a memory configured to store medical images; a user input configured to receive changes from a user; an image processor configured to receive from the user input a change from a first location to a second location for a first segmentation of the medical image, and to use the first patch for the first location and the second patch for the second location for a second segmentation of the medical image, the second segmentation being generated by a machine learning network; a display configured to display a segmentation image of the second segmentation.

[0088] Example embodiment 14: 13. The medical system of example embodiment 13, The medical system, wherein the medical image comprises a magnetic resonance image, an ultrasound image, a computed tomography image, or an X-ray image.

[0089] Example embodiment 15: The medical system of any one of exemplary embodiments 13 to 14, The medical system, wherein the change is a change in the position of a boundary of the first segmentation, the first patch being at the location of the boundary before the change, and the second patch being at the location of the boundary after the change.

[0090] Example embodiment 16: The medical system of any one of exemplary embodiments 13 to 15, 1. A medical system comprising: an image-to-image network, wherein the machine learning network has a first input related to the medical image and a second input related to a relational map that associates positions within the medical image with the first patch and the second patch by similarity.

[0091] Example embodiment 17: 16. The medical system of example embodiment 16, the image processor is configured to generate feature embeddings for the first patch, the second patch, and samples from different positions within the medical image; The relational map is formed from weights that maximize the similarity of the feature embeddings of the patches at different positions to the feature embeddings of the second patches, weighted by the similarity of the feature embeddings of the patches at different positions to the feature embeddings of the first patches.

[0092] Example embodiment 18: 1. A method for artificial intelligence assisted contouring in a medical imaging system, comprising: generating a map representing (a) a similarity of a location in the medical image of a first contour of an object in the medical image to a first position after modification of the first contour, and (b) a dissimilarity of the location to a second position of the first contour of the object in the medical image before the modification; segmenting the object with a machine learning model in response to inputting the medical image and the map into the machine learning model; displaying the segmented object.

[0093] Example embodiment 19: 19. The method of example embodiment 18, comprising: The method, wherein generating the map includes generating the map using embedding features of patches for the location, the first position, and the second position.

[0094] Example embodiment 20: 19. The method of any one of Example Embodiments 18-19, the modification of the first contour is to a portion of the first contour by a user, and other portions of the first contour are not modified by the user; The method, wherein the segmenting includes segmenting with respect to the portion and the other portion without user input for the other portion.

[0095] The various improvements disclosed herein may be used together or separately. Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it will be understood that the invention is not limited to those detailed embodiments, and that various other changes and modifications may be made by those skilled in the art without departing from the scope or spirit of the invention.

Claims

1. 1. A method for assisted segmentation in a medical imaging system, comprising: obtaining an ultrasound image of the patient; segmenting an object in the ultrasound image, the segmentation resulting in a first contour for the object; receiving a change to a portion of the first contour for the object from a user input; extracting positive and negative samples from the ultrasound image based on the modified contour; segmenting the object in the ultrasound image, the segmentation resulting in a second contour by inputting information about the ultrasound image and the positive and negative samples into a first machine learning model; generating an image of the second contour.

2. acquiring the ultrasound image representing a cardiac region of the patient using an intracardiac echocardiographic transducer; The method of claim 1 , wherein the object comprises an ostium, a chamber, or a valve.

3. 2. The method of claim 1, wherein the segmentation resulting in the first contour comprises segmenting by a second machine learning model that outputs the first contour in response to input of the ultrasound image.

4. said receiving including receiving a user input change that shifts a location of a point on said first contour away from said first contour; The method of claim 1 , wherein the portion includes the location and adjacent locations that are shifted in the modification, and other portions of the first contour are unchanged.

5. The extracting step comprises: extracting the positive samples as one or more patches in the ultrasound image that include the portion of the first contour after the modification; The method of claim 1 , comprising extracting the negative samples as one or more patches in the ultrasound image that are separate from the first contour after the modification.

6. The method of claim 1 , wherein the extracting comprises extracting the negative samples as one or more patches in the ultrasound image that include the portion of the first contour before the modification.

7. The method of claim 1 , wherein the segmentation resulting in the second contour comprises segmentation by the first machine learning model including a U-Net.

8. The segmentation resulting in the second contour comprises: forming a contrast weight map from the positive and negative samples; 2. The method of claim 1, further comprising outputting the second contour by the first machine learning model responsive to input of information from the ultrasound image and the contrastive weight map.

9. 9. The method of claim 8, wherein the forming includes generating, for each different location sample from the ultrasound image, a weight from maximizing the similarity of the location sample to the positive samples, inversely weighted by the similarity of the location sample to the negative samples.

10. The method of claim 8 , wherein forming comprises forming the contrastive weight map from embedded features derived from the positive and negative samples by an encoder.

11. The method of claim 8 , wherein the contrastive weight map is based on similarity of locations in the ultrasound image for the positive samples and dissimilarity for the negative samples.

12. The method of claim 1 , wherein the information comprises a relational map formed using the positive and negative samples.

13. 1. An interactive segmentation medical system, comprising: a memory configured to store medical images; a user input configured to receive changes from a user; an image processor configured to receive from the user input a change from a first location to a second location for a first segmentation of the medical image, and to use the first patch for the first location and the second patch for the second location for a second segmentation of the medical image, the second segmentation being generated by a machine learning network; a display configured to display a segmentation image of the second segmentation.

14. The medical system of claim 13 , wherein the medical image comprises a magnetic resonance image, an ultrasound image, a computed tomography image, or an X-ray image.

15. 14. The medical system of claim 13, wherein the change is a change in the position of a boundary of the first segmentation, the first patch being at the location of the boundary before the change and the second patch being at the location of the boundary after the change.

16. 14. The medical system of claim 13, wherein the machine learning network comprises an image-to-image network having a first input related to the medical image and a second input related to a relational map relating positions within the medical image to the first patch and the second patch by similarity.

17. the image processor is configured to generate feature embeddings for the first patch, the second patch, and patches from different positions within the medical image; 17. The medical system of claim 16, wherein the relational map is formed from weights that maximize similarity of the feature embeddings of the patches at different positions to the feature embeddings of the second patches, weighted by similarity of the feature embeddings of the patches at different positions to the feature embeddings of the first patches.

18. 1. A method for artificial intelligence assisted contouring in a medical imaging system, comprising: generating a map representing (a) a similarity of a location in the medical image of a first contour of an object in the medical image to a first position after modification of the first contour, and (b) a dissimilarity of the location to a second position of the first contour of the object in the medical image before the modification; segmenting the object with a machine learning model in response to inputting the medical image and the map into the machine learning model; displaying the segmented object.

19. The method of claim 18 , wherein generating the map includes generating the map using embedding features of patches for the location, the first position, and the second position.

20. the modification of the first contour is to a portion of the first contour by a user, and other portions of the first contour are not modified by the user; The method of claim 18 , wherein the segmenting comprises segmenting with respect to the portion and the other portion without user input for the other portion.

Images