Method and system for ultrasound-based structure positioning using image and user input

Abstract

A method (100) for providing positioning information using an ultrasound system (200), comprising: receiving (140) a plurality of ultrasound images for a first imaged region; receiving (150) a user input via a user interface, the user input comprising location information about the first imaged region; processing (160), using a trained localization algorithm, both (i) one or more of the received plurality of ultrasound images, and (ii) the received user input to generate a localization report for the first imaged region, the localization report comprising one or more of coordinates, contours, and structural identifications; and providing (170) the positioning report to a user via a user interface.

Description

Technical Field

[0001] The present disclosure generally relates to methods and systems for locating anatomical structures using an ultrasound system. Background Art

[0002] Compared to other imaging modalities such as fluoroscopy, computed tomography (CT), and magnetic resonance imaging (MRI), ultrasound offers a low-cost, radiation-free, and real-time means for assessing vascular morphology, topology, and blood flow. Similar advantages of other ultrasound-based imaging procedures are well described in the clinical literature.

[0003] However, today clinical expertise and specialized training are required to identify anatomical structures of interest, detect areas of disease, and obtain accurate measurements during ultrasound scans. For example, in the case of peripheral artery disease (PAD), small limb arteries are particularly difficult to measure. Experienced sonographers often switch back and forth between B-mode and color Doppler modes during a scan. For example, they may scan back and forth while visually tracking an artery through complex anatomy. They may add markers on the screen, or take single-frame snapshots to assist in visual tracking and remember specific locations and landmarks. All of these steps require skills; losing sight of the target during a scan can lead to cumbersome workflows, the need for repeated scans, lengthy procedure times, and reduced procedure results.

[0004] Existing methods for landmark or anatomical feature identification are based on user experience or automatic algorithms. As described above, an experienced sonographer identifies landmarks or anatomical features when viewing ultrasound imaging in real time or via recording, but the process is tedious, time consuming, and error-prone, especially for less experienced sonographers. Algorithmic identification of landmarks or anatomical features is based on a trained algorithm that receives and analyzes one or more ultrasound images for identification. However, algorithmic identification is also an error-prone method.

[0005] US2019 / 099159A1 discloses a method and system for determining measurement points for imaging using a medical scanner.

[0006] US2021 / 224991A1 discloses an artificial intelligence ultrasound medical diagnostic device using semantic segmentation.

[0007] US2021 / 312652A1 discloses an automated prostate analysis system. Summary of the invention

[0008] Therefore, there is a continuing need for methods and systems for improving the positioning of anatomical structures using ultrasound systems.

[0009] The present disclosure relates to creative methods and systems for analyzing ultrasound imaging. Various embodiments and specific implementations herein relate to an ultrasound analysis system, comprising an ultrasound device configured to obtain or otherwise receive multiple ultrasound images for a first imaged area. The system also receives user input via a user interface, the user input including position information about the first imaged area, wherein the user input is derived based on the user's interaction with the ultrasound system when the user obtains multiple ultrasound images for the first imaged area. The system's trained positioning algorithm processes both: (i) one or more of the multiple ultrasound images received, and (ii) the received user input to generate a positioning report for the first imaged area, the positioning report including one or more of coordinates, contours, and structure identifiers. The system then provides the positioning report to the user via the user interface.

[0010] In general, in one aspect, a method for providing localization information using an ultrasound system is provided. The method includes: (i) receiving a plurality of ultrasound images for a first imaged region; (ii) receiving an input, the input including position information about the first imaged region, wherein the input is derived from a user's interaction with the ultrasound system when the user obtains the plurality of ultrasound images for the first imaged region; (iii) using a trained localization algorithm to process both: (1) one or more of the received plurality of ultrasound images; and (2) the received input to generate a localization report for the first imaged region, the localization report including one or more of coordinates, contours, and structure identifiers; and (iv) providing the localization report to a user via a user interface.

[0011] The input derived from the user's interaction with the ultrasound system may be an input corresponding to or derived from an ultrasound acquisition setting, or an input derived from the user's interaction with the ultrasound probe. This recognizes that the user's interaction with the ultrasound system will depend on the anatomical structure being imaged and the field of view of the ultrasound probe, and thus provides indirect user-derived position information that can be used to improve the accuracy and reliability of the output of the trained localization algorithm. By using the input made by the user of the ultrasound system when performing an ultrasound acquisition procedure, the user-derived information can be provided to the trained localization algorithm without the need for additional user input. This reduces the clinician time required to generate a localization report for the imaged area based on the user-derived information, thereby allowing reliable localization information to be provided with improved clinical efficiency.

[0012] Input derived from user interaction with the ultrasound system may include one or more of the following: a center position of the ultrasound probe's field of view when acquiring each image, a scan depth when acquiring each image, a focal depth when acquiring each image, a position and / or orientation of a color Doppler box in each image, a center position of the color Doppler box in each image, a size and / or position of a spectral Doppler gating window within each image, an elasticity / strain map of possible positions for initializing multiple ultrasound images, and / or information derived from a compression manipulation of the ultrasound probe (e.g., changes in image pixel values ​​during a compression manipulation).

[0013] According to an embodiment, the method further comprises: receiving ultrasound scanning information of a current or future scan; and guiding the user to obtain the plurality of ultrasound images for the first imaged area based on the received ultrasound scanning information.

[0014] According to an embodiment, the plurality of ultrasound images are received and processed in real time.

[0015] According to an embodiment, the trained localization algorithm is a segmentation algorithm or an object detection algorithm.

[0016] According to an embodiment, the positioning report further comprises a confidence score generated by said trained positioning algorithm.

[0017] According to an embodiment, the input is derived from one or more ultrasound scans or one or more ultrasound images obtained or received prior to the received plurality of ultrasound images.

[0018] According to an embodiment, the input is an identification of a structure or feature within the first imaged region.

[0019] According to an embodiment, the method also includes: receiving modification input regarding the first imaged area from the user via the user interface in response to the provided positioning report; using a trained positioning algorithm to process both: (i) one or more of the multiple ultrasound images received, and (ii) the received modification input to generate a modified positioning report for the first imaged area; and providing the modified positioning report to the user via the user interface.

[0020] According to another aspect, an ultrasound system is provided, which is configured to provide positioning information. The system includes: a plurality of ultrasound images for a first imaged region; a user interface configured to receive user input, the user input including position information about the first imaged region; a trained positioning algorithm; and a processor configured to use the trained positioning algorithm to process both: (i) one or more of the received plurality of ultrasound images and (ii) the received user input to generate a positioning report for the first imaged region; and direct the user interface to provide the positioning report to the user.

[0021] According to another aspect, a non-transitory computer-readable storage medium has computer-readable program code embodied therein, the computer-readable program code being used to cause an ultrasound system to provide positioning information by: receiving multiple ultrasound images for a first imaged region; receiving input, the input comprising position information about the first imaged region; using a trained positioning algorithm to process both: (i) one or more of the multiple ultrasound images received, and (ii) the received input to generate a positioning report for the first imaged region, the positioning report comprising one or more of coordinates, contours, and structure identifiers; and providing the positioning report to a user via a user interface.

[0022] It should be understood that all combinations of the foregoing concepts and the additional concepts discussed in more detail below (assuming such concepts are not mutually contradictory) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein.

[0023] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the drawings, like reference numerals generally refer to the same parts throughout the different views. Also, the drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

[0025] Figure 1 is a flow chart of a method for providing positioning information using an ultrasound system according to an embodiment.

[0026] Figure 2 is a schematic representation of an ultrasound analysis system according to an embodiment.

[0027] Figure 3 is a flow chart of a method for obtaining ultrasound images and user input according to an embodiment.

[0028] Figure 4 is a flow chart of a method for processing multiple inputs to a positioning algorithm according to an embodiment.

[0029] Figure 5 is a flow chart of a method for providing positioning information using an ultrasound system according to an embodiment.

[0030] Figure 6 is a flow chart of a method for providing positioning information using an ultrasound system according to an embodiment.

[0031] Figure 7 is a flow chart of a method for providing positioning information using an ultrasound system according to an embodiment.

[0032] Figure 8 is a flow chart of a method for training a positioning algorithm according to an embodiment.

[0033] Fig. 9 is a flow chart of a method for modifying a positioning report according to an embodiment.

[0034] Fig.10 is a schematic representation of a positioning report according to an embodiment. DETAILED DESCRIPTION

[0035] The present disclosure describes various embodiments of ultrasound analysis systems and methods. More generally, the applicant has recognized and appreciated that it would be beneficial to provide ultrasound analysis that uses a trained positioning algorithm that receives both user input and ultrasound imaging as input to provide improved feature positioning or identification. For example, an ultrasound analysis system receives or obtains multiple ultrasound images for a first imaged area. The system also receives user input via a user interface, the user input including position information about the first imaged area. The system's trained positioning algorithm processes both: (i) one or more of the multiple ultrasound images received; and, (ii) the received user input to generate a positioning report for the first imaged area, the positioning report including one or more of coordinates, contours, and structure identifiers. The system then provides the positioning report to the user via the user interface.

[0036] Thus, according to an embodiment, the workflow, algorithms and associated user interfaces are intended to facilitate faster ultrasound-based diagnostic and interventional imaging procedures. One example where the present invention may be useful is in assisting peripheral arterial disease (PAD) examinations, where ultrasound is widely used to measure lumen diameter, assess hemodynamic function and support treatment decisions.

[0037] According to an embodiment, ultrasound imaging workflows are simplified by assisting users to quickly locate and track structures that would otherwise be challenging to locate. This can include automatically segmenting the boundaries of small arteries during ultrasound scanning as well as many other ultrasound imaging procedures. The key clinical benefit is that by leveraging user and other input information to support image-based localization, small, challenging structures can be quickly interrogated and relevant anatomical measurements imaged under ultrasound can be more efficiently quantified.

[0038] Ultrasound analysis systems and methods disclosed herein or otherwise contemplated provide numerous advantages over the prior art. An ultrasound analysis system and method is provided that enables improved positioning and / or identification of landmarks or anatomical structures using both ultrasound images and user input, which can improve ultrasound analysis and / or diagnosis, improve patient outcomes, and potentially save lives.

[0039] refer to Figure 1 , in one embodiment, Figure 1 1 is a flow chart of a method 100 for providing positioning information using an ultrasound system. The methods described in conjunction with the accompanying drawings are provided only as examples and should be understood not to limit the scope of the present disclosure. The ultrasound system may be any system described herein or otherwise contemplated. The ultrasound system may be a single system or a plurality of different systems.

[0040] At step 110 of the method, an ultrasound system 200 is provided. Figure 2 , for example, the system includes one or more of a processor 220, a memory 230, a user interface 240, a communication interface 250, a storage device 260, and an ultrasound device 270 interconnected via one or more system buses 212. It will be understood that in some aspects, Figure 2 The system 200 is abstracted, and the actual organization of the components of the system 200 may be different and more complex than that illustrated. In addition, the ultrasound system 200 may be any system described or otherwise contemplated herein. Other elements and components of the system 200 are disclosed and / or contemplated elsewhere herein.

[0041] At step 120 of the method, which is optional, the ultrasound system obtains or receives ultrasound scan information for a current or future scan. The ultrasound scan information may be any information related to the plan, settings, region, or other aspects of the scan. For example, the system first detects or receives information about a particular scan (e.g., a vascular scan), such as via a user preset selection. This information may be received via a user interface, or may be automatically determined by the system when receiving or obtaining settings, parameters, medical commands, or other information about the current or future scan.

[0042] At step 130 of the method, the ultrasound system instructs the user to obtain one or more ultrasound images of the first imaged region based at least in part on the ultrasound scanning information received or obtained by the ultrasound system. For example, the user interface may instruct the user to acquire ultrasound images (e.g., B-mode images) near a structure of interest (e.g., an artery), its upstream branches, and / or its downstream branches. The instructions may be a user interface (UI) floating window that persists until the desired view is achieved, but other methods of providing instructions are also possible.

[0043] At step 140 of the method, the ultrasound system receives or obtains multiple images of a first imaged area of ​​the patient. Depending on the embodiment, the ultrasound image data is sent to the system, obtained by the system, or otherwise received by the system. For example, the ultrasound image data may include a time series of ultrasound image data, such as a video including multiple frames. The ultrasound image data may be obtained for a single imaged area, or may be obtained for multiple areas. The imaged area may be any part of a human or animal body for which ultrasound data may be obtained. The ultrasound image data may be received by the system in real time, or may be stored in local and / or remote memory and received by the system at a future point.

[0044] Any ultrasound device or system may be used to obtain ultrasound image data, which may be any device or system suitable for obtaining or otherwise receiving ultrasound image data of a patient. One or more parameters of the ultrasound device may be set, adjusted, pre-programmed, or otherwise determined by a healthcare professional. The ultrasound device or system may be remote from the ultrasound analysis system 200 or a component thereof.

[0045] At step 150 of the method, the ultrasound system receives user input, the user input including position information about the first imaged region. The user input (also referred to as secondary input) may be received directly from the user via a user interface, or indirectly from the user by analyzing user actions.

[0046] According to an embodiment, a user may select - such as according to an instruction from an ultrasound system or guidance from an ultrasound system - the location of a landmark or anatomical feature / structure or another target in an image. For example, during a procedure, a user may select a landmark or anatomical feature / structure or other target in an image by clicking, tapping or tracing an outline around a structure that he / she is interested in scanning and analyzing. According to an embodiment, the clicked / tapped location is converted into a binary or grayscale image mask, wherein the pixels around the click may be marked as a foreground in the mask. For example, in the case of performing an ultrasound scan in color / power Doppler mode, the system may guide the user to position the Doppler box over (one or more) targets following a standard scanning procedure. The coordinates of the center of the Doppler box (or alternatively, the four corners of the Doppler box) will be converted into an image mask. Thus, according to an embodiment, user input may include direct user actions such as mouse clicks to indicate target (x, y) coordinates.

[0047] According to an embodiment, user input may be derived based on user interaction with the ultrasound system. For example, the system may derive user input based on actions such as user positioning of a color Doppler box, a spectral Doppler gating window on a B-mode image, user placement of a focal depth marker, a user defined scan depth, user centering of the probe / image over a target, and / or user performing a probe compression (or other) manipulation.

[0048] Depending on the embodiment, the user input may be derived from the acquired ultrasound data with or without direct user input. For example, the user input may be derived from: a color / power Doppler signal acquired along with the B-mode image, an image previously collected on the same anatomical structure, an algorithmic prediction from a previous frame of the sequence and converted to position information, and / or other modes of derived position information (such as elasticity or strain), which may supplement the B-mode imagery in addition to the acquired ultrasound data.

[0049] According to an embodiment, in order to utilize user input, the system converts the user input into a format that can be processed by downstream positioning algorithms. For example, the system can convert all input data into the form of an image mask (such as binary or grayscale), where pixels around the location coordinates are marked as foreground in the mask. Figure 3 , in one embodiment, Figure 3 is a flow chart of a method for obtaining ultrasound images and user inputs for use as inputs to a localization algorithm of an ultrasound system. In this embodiment, all secondary inputs are converted to smooth grayscale Gaussian masks, where the Gaussian peaks are centered at position coordinates indicating the location of maximum probability. Figure 3In addition to ultrasound imaging (such as B-mode imaging), the localization algorithm can also use multiple types of user inputs. These secondary inputs can be other ultrasound imaging modes, such as color / power Doppler, user input from the current frame, and / or user or algorithm input from previous frames. The inputs are then combined (such as by cascading all masks into a single input stack) and fed into the localization algorithm.

[0050] Thus, according to an embodiment, the input data is combined and fed into a positioning algorithm, such as by concatenating the masks (along with the B-mode frames) at the outset (ie, early merging), such as Figure 3 Alternatively, each input can be sent to a parallel channel of the network and undergo its own series of processing (convolutions) before being combined inside the network (i.e., post-combination).

[0051] According to another embodiment, all secondary masks are summed or jointly averaged to obtain a single overall mask, which is sent to the localization algorithm, such as Figure 4 As shown. In this method, instead of concatenating the inputs into a stack, the individual masks (probability maps) are summed or co-averaged to get a single overall mask, resulting in a smaller / simpler algorithm or network for localization. One advantage of this approach is that it reduces the amount of processing required by the localization algorithm, allowing for a smaller / simpler network.

[0052] Thus, the ultrasound system may include a user interface that allows user interaction data to be fed directly into the system as input. For example, a user may select a target of interest, such as by clicking on the target. Thus, the user interface may include one or more devices for enabling communication with the user. The user interface may be any device or system that allows information to be transmitted and / or received, and may include a display, mouse, and / or keyboard for receiving user commands.

[0053] In addition to direct user input, the UI may also support one or more ultrasound system controls that can be adjusted by the user and from which the algorithm can indirectly derive position information. One example particularly relevant to vascular localization is user localization of a color Doppler box. For example, the center of the color box can be a secondary position coordinate input into the localization algorithm. Or similarly, the four corners of the color box can be position inputs. Another example is the scan depth and focus depth set by the user when scanning, as these are typically set to match the depth of the target structure(s). A third example is the lateral center of the image, as the user will typically aim to center the ultrasound probe above the target structure(s). A fourth example is user localization of a spectral Doppler gating window. Within this mode, the user defines a small area (gate) within the B-mode image. The center and size of the gate can be converted into position inputs to the algorithm. A fifth example is an elasticity / strain map for initializing possible positions. The system temporarily switches to the appropriate elastic mode for a few frames, gets the initialization data it needs (elasticity / strain maps can be binarized using thresholding to form a direct input to the network), and then switches back to B-mode to use the data in the algorithm. A sixth example could be a compression maneuver performed by the user to differentiate between, say, veins and arteries (veins collapse when compressed, while arteries do not). As the user performs probe compression, the system analyzes the US frames to automatically gather this knowledge. A separate built-in algorithm can look for changes in image pixel values ​​during a compression maneuver.

[0054] Depending on the embodiment, any such user input information may be provided once at the beginning of a scan, and / or updated one or more times during a scan to assist in tracking.

[0055] According to an embodiment, the user interface also displays the results of the algorithm (e.g., predicted positions / contours) in real time. The UI allows manual adjustment of the predictions. The adjusted results are used as corrected input data for subsequent frames, or to reprocess previous frames during retrospective review. Again, this can be done once or multiple times throughout the scan.

[0056] The user interface can be used to reposition the color / power / spectral Doppler box or adjust other settings such as scan depth and focal depth. According to an embodiment, the user interface can be used to make measurements during the scan. For example, when the user scans along the leg, the user may be interested in measuring the diameter or stenosis of a vessel at a specific location. A screen tap or button press will trigger a separate algorithm to calculate the diameter / area / other measurements within the segmented vessels in the current frame. The UI can also prompt the user to make measurements in a specific anatomical region. For example, when a bifurcation is detected based on the segmentation output, the system beeps or shows an indicator on the screen suggesting that the user make measurements in that FOV. When the user stops moving the probe (or freezes the image), the system can then proceed to make (or initialize) those measurements as described herein.

[0057] At step 160 of the method, the trained localization algorithm processes both one or more of the received plurality of ultrasound images and the received user input as input. The trained localization algorithm utilizes the input to generate a localization report for the first imaged region, wherein the localization report includes one or more of coordinates, contours, and structure identifiers. For example, referring to Figure 5 , in one embodiment, Figure 5 is a schematic representation of the workflow of an ultrasound system for positioning. The algorithm processes a sequence of ultrasound images and one or more secondary inputs that give additional position information. The interface allows the generation of secondary inputs, including from user input and user interaction with the system.

[0058] According to an embodiment, the algorithm may be a segmentation algorithm, an object detection algorithm, or other localization algorithm. The algorithm may be a deep neural network model that includes a mechanism for merging multiple different inputs (including user-derived inputs), but other algorithms are also possible. The algorithm is capable of temporal and / or spatial processing to aggregate various forms of secondary input data. This allows the algorithm to focus on the selected (one or more) targets. According to an embodiment, the algorithm outputs the coordinates and / or contours of the detected structure in each frame of the ultrasound sequence. A confidence score may be generated and provided for each prediction.

[0059] It is worth noting that, according to embodiments, not all weights in a neural network are fixed. For example, a spatial attention gate can be used to dynamically generate attention weights based on the image background.

[0060] According to an embodiment, the system may include a specific UI panel that appears at the start of a scan to allow the user to enter all relevant position information (user clicks, Doppler box placement, scan / focus depth, etc.) Various forms of position information may be provided in this manner to initialize the localization algorithm.

[0061] According to an embodiment, the system may be operated with color / power Doppler turned on. The Doppler image may be a second data input, which may be similarly cascaded with the B-mode and mask images. As described above, when Doppler is on, the position of the Doppler box is used as one of the user-derived secondary inputs to the algorithm.

[0062] According to an embodiment, the positioning algorithm allows temporal and spatial processing to aggregate various forms of secondary input data. For example, the algorithm can maintain a memory of previous data, rather than operating on frames independently. This is critical to being able to track (one or more) targets identified in the initial frame throughout the scan. The previous data can be user input information, a previous B-mode / Doppler view, a previous model prediction from a previous frame, or a combination of these. In any case, the previous data is converted into position information as part of the secondary input. The prior information can be continuously updated when the scan is performed based on new incoming frame information. The user can interact with the prior information stored in the memory. For example, if the user adjusts the position / contour predicted by the positioning algorithm, the adjusted result used for subsequent frame predictions is used to improve tracking accuracy.

[0063] Similarly, the user can instruct the system to store specific views that they like during a scan. For example, these may be frames with particularly high image quality, or where the algorithm produces highly accurate results. These snapshot views (or the position information derived at these specific frames) become subsequent secondary inputs into the localization algorithm.

[0064] As part of being able to remember previous data, the localization algorithm can operate on a time series (i.e., a video sequence) as opposed to a single ultrasound frame. For example, the algorithm can include a recurrent neural network architecture that introduces temporal units to achieve time awareness. These units provide a mechanism to encode memory from previous frames and feed this information portion of the secondary input into subsequent frame predictions.

[0065] refer to Figure 6 , in one embodiment, Figure 6 is a flow chart showing one way to incorporate a temporal memory into a deep learning model. In this embodiment, the localization algorithm is a deep neural network that takes a B-mode image and a set (one or more) of secondary inputs. A segmentation network is shown, but the system is not limited to a particular type of network. The example is artery localization, but the system can be applied to other real-time ultrasound imaging use cases.

[0066] According to an embodiment, the localization algorithm incorporates a mechanism for maintaining global spatial information about the image, specifically by aggregating spatial information across all pixels of the image and encoding the global image context. Since all secondary inputs are in the form of position information, efficient aggregation of spatial information is critical. This can be achieved by introducing spatial attention units into the proposed neural network. Feasibility experiments show that these units lead to more accurate segmentation of very small structures that might otherwise be buried within a large number of background pixels in the frame. Figure 6 Describes a mechanism for incorporating spatial attention into the model.

[0067] According to an embodiment, the added potential function of the spatial attention unit is to directly receive various secondary position information. Figure 7 , in one embodiment, Figure 7 is a flow chart of an implementation method of the positioning algorithm. Figure 6 In contrast to the embodiments in , the input is passed to each of the individual "spatial attention units" in the network, rather than at the beginning of the network. The spatial attention units are naturally designed to encode position information. Because the spatial attention units are naturally designed to encode position information, position data can be fed directly to these units, rather than being sent at the beginning of the network (i.e., along with the full B-mode frame). A potential advantage of this approach is that since there are multiple spatial attention units distributed across the layers of the network, this allows more ways to introduce position information throughout the network, including at multiple spatial resolution scales.

[0068] At step 170 of the method, the ultrasound system provides the generated positioning report to the user via a user interface. The positioning report may include one or more of coordinates, contours, and structure identifications, as well as other information. The positioning report may be provided via a user interface using any method for transmitting or displaying information, and the user interface may be any device, interface, or mechanism for providing the transmitted or displayed information. According to an embodiment, the positioning report may be transmitted to another device via wired and / or wireless communication. For example, the system may transmit the report to a mobile phone, a computer, a laptop computer, a wearable device, and / or any other device configured to allow display and / or other communication of the report.

[0069] According to an embodiment, the user interface displays the output of the localization algorithm. These can be on-screen markers / contours, highlighted annotations (e.g., narrowed or bifurcated areas), user guidance elements (e.g., left / right arrows to guide scanning), updated regions of interest (e.g., for automatic Doppler box placement), display of confidence / quality scores, and / or quantitative measurements derived from localization predictions, etc. According to an embodiment, while the user is scanning, the UI allows the user to make real-time adjustments or corrections to the detected positions / contours generated by the localization algorithm. If the user edits the contour on one frame, the results of the manual adjustment are then used to replace the direct algorithm output predicted for subsequent frames to improve tracking.

[0070] According to an embodiment, the localization algorithm outputs the position of the detected object in real time. Additional outputs may include a segmentation mask or a set of coordinates (bounding box, centroid coordinates, radius, overlap fit, etc.). The algorithm may also generate a set of secondary outputs from the temporal and spatial attention units. The output is then used as input for the prediction of the next frame.

[0071] The localization algorithm may additionally output a confidence or quality score to indicate the algorithm's confidence in its predictions. For example, if the target is poorly segmented as determined by the confidence score, the system alerts the user that the target is lost (i.e., the confidence score is below a threshold). All primary or secondary outputs from the algorithm may be interactive with the user and updated in this manner before being sent back as input in subsequent frames to improve tracking accuracy during scanning.

[0072] According to an embodiment, post-processing can be applied to each prediction to refine the predicted vessel contours in each frame, clean up noisy segmented regions, and / or perform frame-to-frame tracking of vessels. The post-processing methods can be spline adaptation, active contour propagation, Kalman / particle filter tracking, or other methods based on classical image processing. Post-processing can also involve user adjustments. If any post-processing is applied, the updated results are used as input for the next frame prediction instead of the original prediction.

[0073] According to an embodiment, sudden changes in the shape or trajectory of target structures can be detected directly from the algorithm output. Most importantly, for vascular clinical applications, this includes vessel bifurcations (when a vessel splits into two or joins into one) and sudden vessel narrowing or flattening (reduction in vessel diameter or cross-sectional area). These events are highlighted to the user so they are aware of the change and can choose to modify the scan or manually adjust the predicted output. Other localization tasks, such as object detection or pose regression, can make use of the above methods and embodiments.

[0074] Thus, according to an embodiment, the UI instructs the user to sweep the probe along the target structure, and the algorithm maintains a memory of past views to track the targets detected in the initial frame while continuously updating. The detection can be continuously updated throughout the sweep. For each frame collected, the output is the location of the detected target structure, the binary segmentation mask of the detected target structure, and the secondary input of the prediction to the next frame. Post-processing can be applied to refine the network output, remove noisy areas, and propagate the predicted contour to the next frame.

[0075] refer to Figure 8 , in one embodiment, Figure 8 8 is a flow chart of a method 800 for training a localization algorithm of an ultrasound system. In step 810 of the method, the system receives a training data set including ultrasound data, preferably for multiple ultrasound scans of multiple imaged regions and multiple patients. The training data may include any information or data required to train the localization algorithm, including ultrasound data that marks or predicts labels of structures or anatomical features, and secondary data such as user input (including any form of user input described or otherwise contemplated herein) that provides additional information about the structures or anatomical features. The training data may be stored in and / or received from one or more databases. The database may be a local and / or remote database. For example, the ultrasound system may include a database of training data.

[0076] According to an embodiment, the ultrasound system may include a data preprocessor or similar component or algorithm configured to process the received training data. For example, the data preprocessor analyzes the training data to remove noise, bias, errors and other potential problems. The data preprocessor may also analyze the input data to remove low quality data. Many other forms of data preprocessing or data point identification and / or extraction are possible.

[0077] At step 820 of the method, the system processes the received information to extract positioning features about the ultrasound data. Positioning features can be any features that will be used to train a positioning algorithm, such as any positioning features that can or will be used for a trained algorithm for positioning analysis for future ultrasound scans. Feature extraction can be implemented by various embodiments for feature recognition, extraction and / or processing, including any method for extracting features from a data set. The result of the feature processing step or module of the ultrasound system is a set of positioning features for multiple ultrasound scans of multiple imaged regions and multiple patients, which therefore comprises a training data set that can be used to train a classifier.

[0078] At step 830 of the method, the system trains a machine learning algorithm, which will be an algorithm for analyzing ultrasound information as described or otherwise contemplated. The machine learning algorithm is trained using the extracted features according to known methods for training machine learning algorithms. According to an embodiment, the algorithm is trained using the processed training data set to generate a localization report based on ultrasound images and user input, as described or otherwise contemplated herein. The generated localization report may include any information described or otherwise contemplated herein, including but not limited to the identification or location of anatomical structures, confidence scores, and other information.

[0079] At step 840 of the method, the trained positioning algorithm is stored for future use. Depending on the embodiment, the trained model may be stored in a local or remote storage device.

[0080] Reference Fig. 9 , in one embodiment, Fig. 9 Is used to modify the Figure 1 A flowchart of a method for providing information in a positioning report generated by a method of the present invention. At step 910 of the method, the ultrasound system receives a modification user input about a first imaged region from a user via a user interface in response to the provided positioning report. For example, a user can review the provided positioning report and identify problems that can be fixed, modified, or otherwise adjusted in the algorithm analysis. The modification user input can be provided using any method for providing user input described or otherwise contemplated herein.

[0081] At step 920, the trained localization algorithm processes the ultrasound image and modifies the user input to generate a modified localization report for the first imaged region. The trained localization algorithm processes the input using any method described or otherwise contemplated herein. The modified localization report may include any information described or otherwise contemplated herein.

[0082] At step 930, the system provides the modification to the user via the user interface. The positioning report can be provided via the user interface using any method for transmitting or displaying information, and the user interface can be any device, interface or mechanism for providing the transmitted or displayed information. According to an embodiment, the positioning report can be transmitted to another device via wired and / or wireless communication. For example, the system can transmit the report to a mobile phone, a computer, a laptop computer, a wearable device, and / or any other device configured to allow display and / or other communication of the report.

[0083] refer to Figure 2 , in one embodiment, Figure 2is a schematic diagram of an ultrasound system 200. The system 200 may be any system described or otherwise contemplated herein and may include any components described or otherwise contemplated herein. It will be understood that in some aspects, Figure 2 An abstraction is constructed, and the actual organization of components of system 200 may be different and more complex than illustrated.

[0084] According to an embodiment, the system 200 includes a processor 220 that can run instructions stored in the memory 230 or the storage device 260 or otherwise process data to, for example, perform one or more steps of the method. The processor 220 can be formed by one or more modules. The processor 220 can take any suitable form, including but not limited to a microprocessor, a microcontroller, multiple microcontrollers, a circuit, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a single processor, or multiple processors.

[0085] Memory 230 can take any suitable form, including non-volatile memory and / or RAM. Memory 230 can include various memories, for example, L1, L2 or L3 cache memory or system memory. Thus, memory 230 can include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM) or other similar memory devices. Among other items, the memory can also store an operating system. The processor uses RAM to temporarily store data. According to an embodiment, the operating system can include code that, when run by the processor, controls the operation of one or more components of system 200. It will be apparent that in an embodiment in which the processor implements one or more of the functions described herein in hardware, the software described in other embodiments as corresponding to such functions can be omitted.

[0086] The user interface 240 may include one or more devices for realizing communication with the user. The user interface can be any device or system that allows to convey and / or receive information, and may include a display, a mouse and / or a keyboard for receiving user commands. In certain embodiments, the user interface 240 may include a command line interface or a graphical user interface, which may be presented to a remote terminal via a communication interface 250. The user interface may be located together with one or more other components of the system, or may be located away from the system and communicate via a wired and / or wireless communication network.

[0087] The communication interface 250 may include one or more devices for implementing communication with other hardware devices. For example, the communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol. In addition, the communication interface 250 may implement a TCP / IP stack for communication according to the TCP / IP protocol. Various alternative or additional hardware or configurations of the communication interface 250 will be apparent.

[0088] The storage device 260 may include one or more machine-readable storage media, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, or a similar storage medium. In various embodiments, the storage device 260 may store instructions executed by the processor 220 or data that the processor 220 may operate on. For example, the storage device 260 may store an operating system 261 for controlling various operations of the system 200.

[0089] It will be apparent that various information described as being stored in storage device 260 may additionally or alternatively be stored in memory 230. In this regard, memory 230 may also be considered to constitute a storage device, and storage device 260 may also be considered to be a memory. Various other arrangements will be apparent. In addition, both memory 230 and storage device 260 may be considered to be non-transient machine-readable media. As used herein, the term non-transient will be understood to exclude transient signals, but rather to include all forms of storage devices, including volatile memory and non-volatile memory.

[0090] Although system 200 is shown as including one of each described component, in various embodiments, each component may be repeated. For example, processor 220 may include multiple microprocessors configured to independently perform the methods described herein or configured to perform the steps or subroutines of the methods described herein, so that multiple processors collaborate to implement the functions described herein. In addition, in the case of implementing one or more components of system 200 in a cloud computing system, various hardware components may belong to separate physical systems. For example, processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.

[0091] According to an embodiment, the storage device 260 of the system 200 may store one or more algorithms, modules and / or instructions to perform one or more functions or steps of the methods described or otherwise contemplated herein. For example, the system may include ultrasound imaging data 262, positioning algorithms 263, and / or reporting instructions 264, among other possible instructions and / or data.

[0092] According to an embodiment, the ultrasound imaging data 262 is any ultrasound imaging data sent to the system, obtained by the system, or otherwise received by the system. The ultrasound image data may include, for example, a time series of ultrasound image data, such as a video including multiple frames. The ultrasound image data may be obtained for a single region, or may be obtained for multiple different regions. The ultrasound image data may be obtained using any ultrasound device or system, which may be any device or system suitable for obtaining or otherwise receiving ultrasound image data of a patient. The ultrasound device or system may be remote from the ultrasound analysis system 200, local to it, or a component thereof.

[0093] According to an embodiment, the localization algorithm 263 is any model or algorithm that is trained or configured to utilize the following two: (i) one or more of the plurality of ultrasound images received. And (ii) the received user input to generate a localization report for the first imaged region. The trained localization algorithm processes the input using any method described or otherwise contemplated herein. The localization report may include any information described or otherwise contemplated herein.

[0094] According to an embodiment, the report instructions 264 direct the system to generate a report or visualization and provide the report or visualization to the user via the user interface 240 of the ultrasound system 200. The positioning report may include any information described herein or otherwise contemplated. Other information is also possible, including but not limited to the identity of the patient, patient demographics, diagnostic or treatment information, and various other possible information. The information can be provided via the user interface using any method for transmitting or displaying information, and the user interface can be any device, interface, or mechanism for providing the transmitted or displayed information. According to an embodiment, the instructions can direct the system to display the information on the user interface or the display of the system. The report can be transmitted to another device via wired and / or wireless communication. For example, the system can transmit the report to a mobile phone, a computer, a laptop, a wearable device, and / or any other device configured to allow other communications of the display and / or report.

[0095] Example

[0096] The following are non-limiting examples of utilization of a localization algorithm for vessel segmentation as described herein or otherwise contemplated.

[0097] According to an embodiment, the methods and systems described herein are implemented as follows Figure 6 The model takes a B-mode video sequence, a color Doppler video sequence, and a user input mask (a button click where the user-selected vessel is converted to a binary mask) as a set of cascaded multi-channel inputs. The primary output is a segmentation mask of the same size as the input image that is tracked throughout the scan. A localization confidence / quality score is also generated for each object in each frame.

[0098] The user interface is important to the entire invention as it allows the user to select a target vessel to track and display automatic vessel diameter measurements and quality / confidence scores. The results of the trained network are presented in Fig.10 It is shown in Fig.10 The vessel segmentation produced by the localization algorithm is shown. From left to right: femoral artery (6 mm diameter), anterior tibial artery (2 mm diameter), and posterior tibial artery (3 mm diameter). The confidence score is calculated by taking the average of the scores of all pixels within the segmented vessel.

[0099] These results indicate the ability of the proposed algorithm to segment different peripheral vessels with a reasonable degree of confidence. The anterior and posterior tibial arteries (ATA and PTA) are among the smallest arteries in the peripheral vasculature that typically must be imaged to assess peripheral vascular health. The impact of secondary inputs, including simulated user click inputs during the scanning sequence, is highlighted in a feasibility experiment performed on an ultrasound sequence acquired during lower limb scans on healthy subjects (Table 1), where the tibia of interest represents <0.1% of the pixels in each frame (2-3mm diameter). The baseline algorithm is a U-Net deep learning segmentation model trained on small tibial arteries (2-3mm diameter) from nine examinations acquired on healthy subjects. The results are independent examinations that are unseen by the model. The impact of secondary inputs, including simulated user click inputs during the scanning sequence, is highlighted.

[0100] As mentioned above, the method and system are not limited to segmentation nor to a specific network architecture. Other localization tasks, such as object detection or pose regression, can also make use of the proposed method.

[0101] Table 1

[0102]

[0103] According to an embodiment, the ultrasound system is configured to process thousands or millions of data points in the input data for training the classifier, as well as process and analyze the received multiple positioning features. For example, using automated processes such as feature recognition and extraction and subsequent training to generate a functional and skillfully trained classifier requires processing millions of data points from the input data and the generated features. This may require millions or billions of calculations to generate novel trained classifiers from millions of data points and millions or billions of calculations. Therefore, based on the input data and parameters of the machine learning algorithm, each trained classifier is novel and different, and thus improves the functionality of the ultrasound system. Therefore, generating a functional and skillfully trained classifier includes a process that the human brain cannot achieve in a lifetime or multiple lifetimes with a certain amount of calculation and analysis.

[0104] In addition, the ultrasound system can be configured to continuously receive localization features, perform analysis, and provide periodic or continuous updates via a report provided to the user for the patient. This requires analyzing thousands or millions of data points on a continuous basis to optimize the report, requiring a certain amount of computation and analysis that the human brain cannot achieve in a lifetime.

[0105] By providing improved ultrasound analysis, this new ultrasound system has a tremendous positive effect on patient care compared to prior art systems. As just one example in a clinical setting, by providing a system that can improve patient analysis and diagnosis with confidence intervals, the system can facilitate treatment decisions and improve survival outcomes, leading to lives saved.

[0106] The present invention may be a system, method and / or computer program product. A computer program product may include a non-transient computer-readable storage medium (or medium) having computer-readable program instructions thereon, for enabling a system or processor to perform aspects of the present invention. A computer-readable storage medium may be a tangible device that can save and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the foregoing, as well as other possibilities. The computer-readable program instructions described herein may be downloaded to a corresponding computing / processing device from a computer-readable storage medium or an external computer or an external storage device via a network (e.g., the Internet, a local area network, and / or a wireless network). Aspects of the present invention are described with reference to flowcharts and / or block diagrams of methods, devices, systems, and computer program products according to embodiments of the present invention. It will be understood that each box of a flowchart illustration and / or a block diagram and a combination of boxes of a flowchart illustration and / or a block diagram may be implemented by computer-readable program instructions.

[0107] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0108] Unless expressly indicated to the contrary, the terms "a" and "an" as used in the specification and claims herein should be understood to mean "at least one."

[0109] As used in the specification and claims herein, the phrase "and / or" should be understood to mean "either or both" of the elements so combined, i.e., elements that are present in combination in some cases and separately in other cases. Multiple elements listed with "and / or" should be interpreted in the same manner, i.e., "one or more" of the elements so combined. In addition to the elements specifically identified by the "and / or" clause, other elements may optionally be present, whether related or unrelated to those elements specifically identified.

[0110] "Or" as used in the specification and claims herein should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, that is, including at least one, but also including more than one element of a plurality of elements or a list of elements and optionally additional unlisted items. Only terms that clearly indicate the opposite, such as "only one of..." or "exactly one of...", or, when used in the claims, "consisting of..." will refer to including exactly one element of a plurality of elements or a list of elements. Generally, the term "or" as used herein should be interpreted as indicating an exclusive alternative (i.e., "one or the other but not two") only when it is preceded by an exclusive term (e.g., "any one of...", "one of...", "only one of...", or "exactly one of...").

[0111] The phrase "at least one" as used in the specification and claims herein referring to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one element of each and every element specifically listed in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows that elements may optionally be present in addition to the elements specifically identified in the list of elements to which the phrase "at least one" refers, whether related or unrelated to those elements specifically identified.

[0112] It should also be understood that in any method claimed herein that includes more than one step or action, the order of the steps or actions of the method is not necessarily limited to the order in which the steps or actions of the method are described unless explicitly stated to the contrary.

[0113] In the claims and the foregoing specification, all transitional phrases, such as "comprises," "comprising," "carrying," "having," "containing," "involving," "holding," "including," etc., shall be understood to be open-ended, i.e., meaning including but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases, respectively, as explained in Section 2111.03 of the Manual of Patent Examining Procedure of the United States Patent Office.

[0114] Although several inventive embodiments have been described and illustrated herein, it will be readily apparent to those of ordinary skill in the art that various other modules and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein will be considered to be within the scope of the inventive embodiments described herein, and each of such variations and / or modifications is considered to be within the scope of the inventive embodiments described herein. More generally, it will be readily understood by those skilled in the art that all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and that actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications using the invention teachings. Those skilled in the art will recognize or be able to determine many equivalents of the specific inventive embodiments described herein using experimental means that do not exceed conventional means. Therefore, it should be understood that the foregoing embodiments are presented only by way of example, and within the scope of the claims and their equivalents, the inventive embodiments may be practiced in a manner different from that specifically described and claimed. The inventive embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the invention of the present disclosure.

Claims

1. A method (100) for providing positioning information using an ultrasound system (200), comprising: receiving (140) a plurality of ultrasound images for a first imaged region; receiving (150) an input, the input comprising position information about the first imaged region, wherein the input is derived based on an interaction of the user with the ultrasound system when the user obtains the plurality of ultrasound images for the first imaged region; Using the trained localization algorithm to process (160) both: (i) one or more of the plurality of received ultrasound images, and (ii) the received input, to generate a localization report for the first imaged region, the localization report comprising one or more of: coordinates, contours, and structure identifications; and The location report is provided (170) to the user via a user interface.

2. The method according to claim 1, further comprising the steps of: receiving (120) ultrasound scan information for a current or future scan; and The user is guided (130) to obtain the plurality of ultrasound images for the first imaged region based on the received ultrasound scanning information.

3. The method according to claim 1, wherein: The multiple ultrasound images are received and processed in real time.

4. The method according to claim 1, wherein: The trained localization algorithm is a segmentation algorithm or an object detection algorithm.

5. The method according to claim 1, wherein: The positioning report also includes a confidence score generated by the trained positioning algorithm.

6. The method according to claim 1, wherein: The input is also derived from one or more ultrasound scans or one or more ultrasound images obtained or received prior to the received plurality of ultrasound images.

7. The method according to claim 1, wherein: The input is an identification of a structure or feature within the first imaged region.

8. The method according to claim 1, further comprising: In response to the provided positioning report, receiving (910) modification input from the user via the user interface regarding the first imaged region; using the trained localization algorithm to process (920) both: (i) one or more of the received plurality of ultrasound images, and (ii) the received modification input to generate a modified localization report for the first imaged region; as well as The modified position report is provided (930) to the user via a user interface.

9. An ultrasound system (200) configured to provide positioning information, comprising: a plurality of ultrasound images (262) of a first imaged region; a user interface (240) configured to receive user input, the user input comprising position information about the first imaged region, wherein the user input is derived based on an interaction of the user with the ultrasound system when the user obtains the plurality of ultrasound images for the first imaged region; trained localization algorithm (263); A processor (220) is configured to: use the trained localization algorithm to process both: (i) one or more of the received multiple ultrasound images, and (ii) received user input to generate a localization report for the first imaged area; and direct the user interface to provide the localization report to the user.

10. The ultrasound system of claim 9, wherein: The positioning report includes one or more of the following: coordinates, contours, and structure identification.

11. The ultrasound system of claim 9, wherein: The positioning report also includes a confidence score generated by the trained positioning algorithm.

12. The ultrasound system of claim 9, wherein: The user input is derived from one or more ultrasound scans or one or more ultrasound images obtained or received prior to the received plurality of ultrasound images.

13. The ultrasound system of claim 9, wherein: The user input is an identification of a structure or feature within the first imaged region.

14. A non-transitory computer readable storage medium (260) having computer readable program code embodied therein, the computer readable program code being configured to cause an ultrasound system (200) to provide positioning information by: receiving a plurality of ultrasound images for a first imaged area; receiving an input, the input comprising position information about the first imaged area, wherein: deriving the input based on an interaction between the user and the ultrasound system when the user obtains the plurality of ultrasound images for the first imaged area; Using the trained localization algorithm to process both: (i) one or more of the plurality of received ultrasound images and (ii) the received input to generate a localization report for the first imaged region, the localization report comprising one or more of: coordinates, contours, and structure identifiers; and The positioning report is provided to the user via a user interface.

Images