Automated Needle Detection

Abstract

SOLUTION: In one embodiment, an ultrasound imaging system is configured to receive a set of ultrasound images of a target anatomical region. The set of ultrasound images is combined to create a composite tissue frame. The ultrasound imaging system determines whether an interventional instrument is present within the ultrasound images based on a set of trained classification algorithms based on the set of ultrasound images and the composite tissue frame. If an interventional instrument is detected, the ultrasound imaging system further determines whether an additional ultrasound frame should be captured to image the interventional instrument, and the steer angle to be used for the additional ultrasound image. The ultrasound imaging system determines a linear structure corresponding to the interventional instrument, and creates a blended image showing the interventional instrument and the composite tissue frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Cross-reference to related applications This application is related to and claims the benefit of U.S. patent application Ser. No. 16 / 210,253, filed Dec. 5, 2018, which is incorporated by reference in its entirety herein.

[0002] Technical Field The disclosed technology relates to ultrasound imaging systems, and more particularly to ultrasound imaging systems for imaging interventional instruments within the body. [Background technology]

[0003] Ultrasound imaging is becoming increasingly accepted as a standard of care to be used when guiding an interventional instrument to a desired location within the body. One common use for this procedure is during the administration of anesthesia, whereby a physician or medical technician views an ultrasound image to help guide a needle to a desired nerve or area of ​​interest. To enhance the physician's ability to see the needle, many ultrasound systems can incorporate so-called "needle visualization" techniques that generate a composite image from an anatomical image and an image of the needle. To obtain the best image, the direction of the transmit beam can be approximately perpendicular to the needle or other interventional instrument. In a specific embodiment, the user or ultrasound system may take several images at different angles to vary the beam direction, so that the best image of the needle can be identified and used. Summary of the Invention

[0004] In specific embodiments, existing needle detection systems may require the user to provide at least two inputs: 1) an indication that needle detection should be used; and 2) a direction in which the needle enters the image frame (e.g., left or right entry). In certain embodiments, if the user does not indicate that needle detection should be used, no needle detection image is captured. If the user selects an incorrect needle entry direction, the transmit beam will not intersect the needle, and so the needle will not be detected. In certain embodiments, another potential problem with existing needle detection systems is that multiple needle visualization frames must be captured (saved), which may reduce the overall frame rate of the composite image. In certain embodiments, if needle entry information can be determined from the current set of frames, then it may not be necessary to acquire additional needle visualization frames, thereby improving the frame rate of ultrasound imaging. In certain embodiments, the needle detection system may detect an interventional instrument based on tissue warping or temporal changes in an image of the biological tissue, without relying on needle visualization frames to conclude that an interventional instrument is present. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 shows a simplified diagram of an exemplary ultrasound imaging system for generating and displaying mixed images of tissue and interventional instruments. [Diagram 2] FIG. 2 illustrates a block diagram of an example ultrasound imaging system. [Diagram 3] FIG. 3 is a diagram showing an example of multiple ultrasound images taken at multiple angles to identify the most appropriate angle for the needle frame. [Figure 4] FIG. 4 shows an exemplary diagram of an ultrasound imaging system capable of generating and displaying mixed images of tissue and interventional instruments. [Diagram 5]FIG. 5 shows an exemplary diagram of an ultrasound imaging system that can automatically select an optimal needle frame from multiple steering angles and generate and display a mixed image of tissue and interventional tools. [Figure 6A] FIG. 6A shows an example needle frame image that is combined with a tissue frame to create a blended image for display. [Figure 6B] FIG. 6B shows an example needle frame image that is combined with a tissue frame to create a blended image for display. [Figure 6C] FIG. 6C shows an example needle frame image that is combined with a tissue frame to create a blended image for display. [Figure 7] FIG. 7 shows an exemplary diagram of an ultrasound imaging system that can automatically detect the presence and direction of entry of an interventional instrument, automatically select the optimal needle frame from multiple steering angles, and generate and display a mixed image of tissue and interventional instrument. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As described in further detail below, the disclosed technology relates to improvements in ultrasound imaging systems, and in particular to ultrasound imaging systems configured to automatically detect the presence and direction of entry of an interventional instrument inserted in imaged tissue and generate a combined image of the tissue and the interventional instrument. In the following description, the interventional instrument is described as being a needle used to deliver anesthesia or other drugs to a desired location. However, other devices such as biopsy needles, needles for suturing tissue, needles for withdrawing fluids (e.g., amniocentesis), robotic surgical instruments, catheters, guidewires, or other invasive medical instruments may be imaged as well.

[0007] In one embodiment, a processor within the ultrasound system may be configured to generate and deliver several transmit beams to the body to generate an image of the tissue (anatomy) under examination. In addition, the processor may be configured to generate several transmit beams at different transmit beam angles, i.e., steering angles, to image an interventional instrument. To distinguish the image of the tissue from the image of the interventional instrument, the frame of the interventional instrument is sometimes referred to as a "needle frame" even though the instrument is not a needle.

[0008] Each of the needle frames generated from transmissions at different transmit angles may be analyzed to detect the presence of an interventional instrument. In one embodiment, a composite image may be generated using echo data from one or more needle frames captured using different transmit beam directions and images of the biological tissue to indicate both the location of the tissue and the interventional instrument. In certain embodiments, by capturing images of the biological tissue and multiple images of the needle frames over time, the ultrasound imaging system can provide a live view of the composite image to a user, such that the user can observe the location of the interventional instrument relative to the structure of the biological tissue over time.

[0009] 1 illustrates an exemplary ultrasound imaging system that implements the disclosed techniques for imaging tissue of a patient. In certain embodiments, ultrasound imaging system 10 can be a handheld, portable, or cart-based system that uses a transducer probe 12 to transmit ultrasound signals to a region of interest and receive corresponding echo signals to generate an image of the tissue being scanned. Probe 12 can be a one- or two-dimensional linear or curved transducer, or a phased array transducer, all of which can have their steering angle selectively changed electronically.

[0010] The ultrasound imaging system 10 can convert the characteristics of the received echo signals (e.g., their amplitude, phase, power, frequency shift, etc.) into data that can be quantified and displayed for a user as an image. The generated images can be stored electronically for digital record keeping or transmitted via a wired or wireless communication link to another device or location. In certain embodiments, an operator can hold the probe 12 with one hand while guiding an interventional instrument 15 into a patient (or subject) 20 with the other hand. The ultrasound imaging system 10 can automatically detect the presence of the interventional instrument 15 and identify one or more needle frames that best depict the location of the interventional instrument 15. The operator can view a composite image 22 of the tissue and an indication 24 of where the interventional instrument is located within the tissue. The composite image 22 can be updated on the screen while the instrument is being guided to the target location. Such a location can be a particular nerve site in the field of anesthesia, or other area of ​​interest such as a blood vessel or a particular organ (e.g., uterus, prostate, tumor, cardiovascular, etc.).

[0011] As can be appreciated by one of ordinary skill in the art, the optimal beam direction for imaging a long, thin interventional instrument (such as a needle) may be at an angle that is approximately perpendicular to the length of the instrument. However, the imaging parameters and beam directions required to image the instrument are often not the same as those optimal for imaging the tissue. In certain embodiments, the user may not need to select a particular beam angle to use in generating the needle frame. Instead, the processor may be programmed to generate the needle frame using multiple different steering angles. The echo data of the needle frames generated from these different steering angles may be analyzed to detect the presence of an object that may be an interventional instrument. In certain embodiments, the detection of the interventional instrument and the identification of the best needle frame steering angle for visualization of the interventional instrument may be identified via a trained machine learning algorithm. By way of example and not limitation, a trained neural network may analyze one or more ultrasound images to identify whether an interventional instrument is present and the appropriate angle for capturing the interventional instrument so that the needle frame is combined with the image of the biological tissue to generate a composite frame. In certain embodiments, echo data potentially representative of the interventional instrument from one or more needle frames acquired using different transmit beam directions may be copied from the needle frame and fused with the echo data for the image of the biological tissue to generate a composite image showing both the tissue and the location of the interventional instrument.

[0012] FIG. 2 shows a simplified block diagram of an ultrasound imaging system according to one embodiment of the disclosed technology. In certain embodiments, the ultrasound system may be configured with different components than those shown. In addition, the ultrasound system may include parts that are not described (e.g., power supplies, etc.) and are not necessary to understand how to make and use the disclosed technology. In the illustrated embodiment, the ultrasound system may include a processor 40, which may have an internal or external memory (not shown) that includes instructions executable by the processor to operate the ultrasound imaging system as described in detail below. In the transmit path, the ultrasound system may include a transmit beamformer 42, a transmit gain control amplifier 44, and a transmit / receive switch 46. If the ultrasound probe 12 is of a phased array type or is otherwise capable of electronically changing the transmit angle, the transmit beamformer 42 may be operative to generate a number of signals having relative amplitudes and phases (timing) selected to generate an ultrasound beam from some or all of the transducer elements (the transducer elements of the probe) that are structurally reinforced in a desired transmit beam direction (desired steering angle). The signals from the transmit beamformer may be amplified by the transmit amplifier 44 to a voltage level high enough to cause the transducer elements to produce a desired acoustic signal in the tissue being examined. In certain embodiments, the processor 40 may be connected to provide a control command, such as a digital value (e.g., 0 to 255), to the gain control amplifier. The value of the command may control the amount of gain provided by the transmit amplifier 44.

[0013] Other techniques for adjusting the power of the ultrasound signal may include varying the waveform that drives the transducer elements to increase or decrease the power of the ultrasound signal. In certain embodiments, the voltage rails (+V, -V) of the amplifier that generates the drive signal may be altered to change the power of the ultrasound signal. In certain embodiments, the drive signal may be delivered to fewer or more transducer elements to change the power of the ultrasound signal. Those skilled in the art will appreciate that these techniques are merely exemplary and that there may be many ways in which the level of acoustic power of the ultrasound signal delivered to the patient may be adjusted.

[0014] In certain embodiments, the amplified transmit signal can be provided to the transducer probe 12 through a transmit / receive switch 46 that disconnects or shields sensitive receive electronics from the transmit signal at the time the transmit signal is provided to the transducer probe 12. After the signal is transmitted, the transmit / receive switch 46 can connect the receive electronics to the transducer elements for sensing corresponding electronic echo signals generated when returning sound waves impinge on (strike) the transducer elements.

[0015] In certain embodiments, in the receive path, the ultrasound imaging system may include a low noise amplifier 50, a time gain control amplifier 52, an analog-to-digital converter 54, a receive beamformer 56, and an image processor 58. Analog echo signals generated by the imaging probe may be directed via the transmit / receive switch 46 to the low noise amplifier where the analog echo signals are amplified. The TGC amplifier 52 may apply variable amplification to the received signal, varying the applied amplification level with the return time of the signal (e.g., proportional to the depth within the tissue being imaged) to combat signal versus depth attenuation. The amplified signal may then be converted to a digital format by the analog-to-digital converter 54. The digitized echo signals may then be delayed and summed by a receive beamformer 56 before being provided to an image processor.

[0016] In certain embodiments, the number of transmitted beams (lines) and received beams (lines) may differ from each other. By way of example and not limitation, the receive beamformer may generate two or more adjacent lines per transmit beam in parallel (i.e., simultaneously), a technique sometimes known as parallel receive beamforming or multiline processing. Multiline processing may be used to increase the imaging frame rate by lowering the number of transmit beams while the number of receive lines per frame (line density) can be kept constant. In certain embodiments, a higher multiline order (the number of receive lines beamformed in parallel from a single transmit beam) can be used to increase the number of receive lines per frame while keeping the number of transmit beams, and hence the frame rate, constant. Other combinations of line density, frame rate, and multiline order are possible as well. Furthermore, it may be possible to transmit an unfocused beam (plane wave) and beamform all receive lines of a frame from that single transmit beam. The system may also employ different combinations of multiline order and line density for imaging tissue versus imaging an interventional instrument. In certain embodiments, while improving frame rate, using a higher multiline order, lower line density, or a non-focused transmit beam may degrade the quality of the acquired image.

[0017] Images generated by the image processor 58 from the received signals may be displayed on a display 60. Additionally, the images may be stored in an image memory (not shown) for future recall and review. Several inputs 72 may be provided to allow an operator to change various operating parameters of the ultrasound imaging system as well as input data such as a patient's name or other record-keeping data. Additionally, the ultrasound imaging system may include input / output (I / O) circuitry that allows the system to connect to a computer communications link (LAN, WAN, Internet, etc.) via a wired (e.g., Ethernet, USB, Thunderbolt, Firewire, or the like) or wireless (802.11, cellular, satellite, Bluetooth, or the like) communications link.

[0018] The details of the components that comprise an ultrasound imaging system and how they operate can generally be considered to be known to those of ordinary skill in the art. Although the ultrasound imaging system is shown as having many separate components, devices such as ASICs, FPGAs, digital signal processors (DSPs), CPUs, or GPUs can be used to perform the functions of more than one of these individual components.

[0019] As described above, the processor 40 may be programmed to generate a composite image of the tissue being examined and the interventional instrument being introduced into the tissue. In certain embodiments, the image processor generates an anatomical image of the tissue being examined using imaging parameters selected for the depth and particular type of tissue being scanned. The anatomical image generated by the image processor 58 can be stored in memory to be combined with echo data for one or more needle frames generated to locate the position of the interventional instrument.

[0020] In one embodiment, the processor can cause the transmit electronics to generate transmit beams in several different transmit directions to image the interventional instrument. By way of example and not limitation, the processor 40 can direct the transmit beams to be generated at shallow, medium, and steep angles as measured relative to the longitudinal axis of the transducer probe. In certain embodiments, the location of the interventional instrument can appear more clearly in one or more needle frames than the rest. Exemplary needle visualization techniques are shown in commonly owned U.S. Patent Application Serial No. 15 / 347,697, filed November 9, 2016, and incorporated herein by reference.

[0021] In certain embodiments, the echo data for each needle frame generated from transmissions at different transmit angles may be analyzed for the presence of an interventional instrument. Various instrument detection algorithms may be used. By way of example and not limitation, the image may be analyzed for the presence of a linear segment of pixels that are much brighter (e.g., of greater amplitude) than adjacent pixels, thereby indicating the presence of a strong linear reflector. The length of the segment that may represent the interventional instrument may vary, and in some embodiments may be curved, if the interventional instrument itself is curved or bends as the instrument is inserted. Alternatively, the segment may appear curved in the coordinate system in which detection is performed if the image was acquired with a curved transducer geometry (e.g., convex).

[0022] In certain embodiments, each segment of bright pixels is scored to indicate the likelihood that the segment represents an interventional device. By way of example and not limitation, such scores may be adjusted depending on the length of the bright pixels over a certain threshold, how straight or linear the segment of pixels is, how much contrast exists between the bright pixels and adjacent pixels, how strong the edges around the segment of bright pixels are as identified by gradient or other edge detection operations, etc. A Hough transform or other similar technique can be used to identify the locations of pixels that lie on linear or parameterized curved segments, and a score may then be determined.

[0023] In certain embodiments, the brightness data values ​​for the image can be converted to corresponding gradient values ​​by looking at the difference between adjacent brightness values ​​along the beamline. An interventional instrument, a needle or other bright reflector, can generally be characterized as a large negative gradient of brightness values ​​(e.g., light to dark) closely followed by a large positive gradient of brightness values ​​(e.g., dark to light) when viewed in the direction from the transducer and into the tissue. The gradient values ​​can be filtered to ensure that the large changes in positive and negative gradient values ​​occur within the distance expected for the interventional instrument. A Hough transform can then be used to identify whether the large positive / negative gradient changes occur in a linear pattern in adjacent beamlines. The score for a segment of a large gradient change can be increased or decreased depending on one or more of the length of the gradient change, how close the positive gradient change is to the negative gradient change, and how the gradient changes align spatially from beam to beam.

[0024] In certain embodiments, segments of echo data can be scored according to how large the gradient is and how well the gradient aligns in adjacent beamlines. Such segments that are more aligned together with larger gradients can be given a higher score than segments that have smaller gradients and are less aligned. A representation of an instrument can include a single long segment or multiple short segments, and not all of the segments with the highest scores originate from the same needle frame.

[0025] In certain embodiments, echo data from the images with the highest scored segments that likely represent an interventional instrument is copied from two or more needle frames and the echo data is blended with the anatomy image. In another embodiment, echo data may be copied from a single needle frame and the echo data may be blended with echo data for the anatomical image. In certain embodiments, other needle visualization techniques may also be used to detect the presence of a needle or other interventional instrument in an ultrasound image.

[0026] In certain embodiments, the processor can be programmed to identify segments of pixel data from one or more needle frames generated from transmissions taken at various transmit angles, where these segments have a score indicating that the pixels likely represent an interventional instrument, and the processor can use a blending function to copy the pixel data representing the instrument and combine the copied pixel data with pixel data in the tissue image.

[0027] In certain embodiments, pixel data used to display an ultrasound image may be analyzed to detect the presence of an interventional instrument. In certain embodiments, echo data is analyzed prior to conversion to pixel data ready for display. By way of example and not limitation, echo data that has been amplified, converted to digital, and beamformed but not yet scan converted may be analyzed to detect and score segments within the data that are representative of an interventional instrument.

[0028] In certain embodiments, generating several needle frames at different transmit angles may reduce the frame rate of the ultrasound imaging system. In certain embodiments, the processor may select a higher line density and / or a lower multi-line setting for a subset of needle frames (e.g., multiple high quality needle frames) than the rest of the needle frames (low quality needle frames), where the high quality needle frames are adaptively selected with a high score (i.e., a high probability for the presence of an interventional instrument) based on the orientation of the structures detected by the instrument detection algorithm. If the detection algorithm finds structures of similar orientation with a high score in multiple needle frames, the needle frame with an imaging steering angle that would insonify the interventional instrument at an angle close to perpendicular to the probe orientation may be selected to become the acquisition of the high quality needle frame, and the other needle frames may continue to be acquired as lower quality needle frames. Adaptive selection of a high quality needle frame can also ensure high quality visualization of the "most likely" interventional instrument and the remaining low quality needle frames can ensure that instruments in other orientations are not overlooked and that the operator may not be required to manually select a high quality needle frame if the angle of the interventional instrument relative to the ultrasound scan head changes during processing.

[0029] In certain embodiments, the system may adaptively change the employed angles based on the detection score, as well as the acquisition rate and number of needle frames. By way of example and not limitation, if a feature with a high detection score is identified in a needle frame with a particular angle setting, the system may designate that angle as a "high priority angle" and increase the acquisition rate of needle frames at or near that angle setting, while slowing down the acquisition rate of needle frames at angles farther away from the high priority angle or at angles for needle frames that do not contain features with high segment scores. In certain embodiments, the system may continue to acquire and analyze needle frames with angles farther away from the high priority angle setting as "scouting" needle frames, so that the system can re-evaluate and change the "high priority angle" on the fly if a feature with a higher detection score is detected any time in those scouting frames. In certain embodiments, the angles of the scouting needle frames may be selected to have a larger angle spread between them and / or a lower acquisition rate on the frame rate to minimize the overall impact.

[0030] FIG. 3 illustrates an example of three needle frame images captured by an ultrasound imaging system. In the example of FIG. 3, the needle frame images may be captured at a shallow angle, a medium angle, and a steep angle. In certain embodiments, each of the needle frame images may be at a steeper angle than the image captured normally for the B-mode frame. In each of the three needle frames depicted in FIG. 3, an interventional instrument (e.g., a needle) may be visible within the frame. In FIG. 3, the needle frame captured at the shallow angle may show the needle more clearly compared to the medium and steep angles. In this example, the needle is perpendicular to the angle of the transmit beam in the shallow frame, while it may be seen that the needle is not completely perpendicular to the angle of the transmit beam in the medium and steep frames. In certain embodiments, the angle used for the needle frame images may be too steep to generate an image of the structure of the biological tissue within the field of view of the ultrasound image.

[0031] FIG. 4 illustrates an exemplary system for generating a blended image that combines a linear structure corresponding to an interventional instrument (e.g., a needle) with a composite tissue frame. In the example of FIG. 4, an ultrasound imaging system can capture three frames 410, 411, and 412 to be used to identify the tissue frame. In certain embodiments, the three frames 410, 411, and 412 are taken with different transmit beam directions but directed at the same anatomy. By way of example and not limitation, frame 410 is taken at an angle of −14 degrees, frame 411 is taken at an angle of 0 degrees, and frame 412 is taken at an angle of +14 degrees. In certain embodiments, the angles used for frames 410, 411, and 412 may be shallower than the angles required to visualize the interventional instrument. In certain embodiments, the angles used for frames 410, 411, and 412 are angles suitable for ultrasound images in intensity mode, i.e., B-mode. By way of example and not limitation, to generate a B-mode image to reduce speckle, three or five image frames may be captured, with at least one frame captured at 0 degrees and the other frames captured between -15 degrees and +15 degrees. In step 420, the ultrasound imaging system combines frames 410, 411, and 412 to generate a composite tissue frame 430 that depicts the structure of the biological tissue within the field of view. In certain embodiments, frames 410, 411, and 412 are image frames suitable for one or more other imaging modes, such as a color mode. In certain embodiments, frames 410, 411, and 412 may be reconfigured over time such that frames suitable for a first imaging mode are interleaved with frames suitable for a second imaging mode.

[0032] In certain embodiments, the ultrasound imaging system may also capture a needle frame 440, which may be captured at a steeper angle to enhance visualization of the interventional instrument. In certain embodiments, a user of the ultrasound imaging system may manually select the option to capture one or more needle frames. In such embodiments, the user may input whether to angle the image to the left or right, and the user may input the steering angle of the needle frame. In the example of FIG. 4, the ultrasound image frame 440 may be captured at a +30 degree angle, which may be more suitable for reflecting the ultrasound signal away from the shaft of the interventional instrument. In step 442, the ultrasound imaging system determines whether a structure is present, and in step 444, determines whether the imaging probe is straight or curved, and if the probe is curved, then the ultrasound imaging system performs a scan conversion in step 446. In step 450, the ultrasound imaging system may determine the presence of a linear structure corresponding to the interventional instrument and its location in the ultrasound image. The ultrasound imaging system can then generate an image 460 of the linear structure depicting the interventional tool using a masked region defined about the linear structure in the same field of view as the composite tissue frame 430. In step 470, the ultrasound imaging system can merge the composite tissue frame 430 with the linear structure 460 in the masked region to generate a blended image 480 depicting the enhanced interventional tool and its relative position to the imaged anatomy.

[0033] FIG. 5 shows another exemplary system for generating a blended image that combines a linear structure corresponding to an interventional instrument with a composite tissue frame. In the example of FIG. 5, the ultrasound imaging system can capture multiple needle frames and identify which frame is most suitable for combining into a blended image. In FIG. 5, similar to FIG. 4, the ultrasound imaging system can capture three frames 510, 511, and 512 at multiple angles suitable for combining into a B-mode image. In step 520, the ultrasound imaging system stitches the frames 510, 511, and 512 into a composite tissue frame 530 for spatial blending. Upon receiving an indication from the user that an interventional instrument is present and the direction in which the interventional instrument is being introduced, the ultrasound imaging system can capture multiple needle frames 540, 541, and 542 from different angles. 5, needle frame 540 is captured at a steering angle of +24 degrees, needle frame 541 is captured at a steering angle of +32 degrees, and needle frame 542 is captured at a steering angle of +40 degrees. In certain embodiments, the angles of the shallow, medium, and steep needle frames may be pre-specified by the ultrasound imaging system or may be pre-set by a user.

[0034] In the example of FIG. 5, the ultrasound imaging system then identifies which of the needle frames 540, 541, 542 are suitable for identifying linear structures. In certain embodiments, the ultrasound imaging system identifies which needle frame is most indicative of the presence of an interventional instrument by selecting the frame with the highest linear structure score. In step 543, the ultrasound imaging system identifies the presence of a structure in one or more needle frames and in step 544, identifies whether the imaging probe is linear or curved. If the probe is curved, in step 546, the ultrasound imaging system performs a scan conversion. In step 550, the ultrasound imaging system can identify the presence of linear structures corresponding to the interventional instrument and their location in the ultrasound image. In certain embodiments, the ultrasound imaging system can identify which of the needle frames 540, 541, and 542 should be used for enhancing linear structures and can remove the non-selected needle frames from consideration. The ultrasound imaging system may then generate an enhanced linear structure image 560 depicting the interventional instrument (at the selected needle frame angle) using a masked region defined around the linear structure in the same field of view as the composite tissue frame 530. In step 570, the ultrasound imaging system may merge the composite tissue frame 530 with the enhanced linear structure 560 to generate a blended image 580, which depicts the interventional instrument and its relative position to the structure of the imaged anatomy.

[0035] In certain embodiments, an advantage of the system of Figure 5 over a system such as that depicted in Figure 4 may be that the user does not need to specify a steering angle for depicting line structures, and because multiple needle frames are used, the ultrasound imaging system can dynamically identify the best angle for depicting the interventional instrument over time and can switch the needle frame angle used as needed. In certain embodiments, a potential disadvantage to the system of Figure 5 is that because three needle frames must be captured for each set of composite tissue frames (which are themselves a combination of three frames), the system of Figure 5 may require at least six total frames to be captured to generate a single blended image, which may reduce the frame rate of the ultrasound imaging system.

[0036] 6A-6C show exemplary images of needle frames and composite tissue frames that can be combined into a blended image by an ultrasound imaging system. FIG. 6A depicts a needle frame, where a linear structure corresponding to an interventional instrument is visible in the center of the frame. In certain embodiments, the ultrasound imaging system can identify the extent of the interventional instrument within the needle frame. By way of example and not limitation, an x-coordinate corresponding to the left-most or right-most portion of the linear structure may be identified. FIG. 6B depicts a composite tissue frame depicting tissue structures present in the same field of view as depicted in FIG. 6A. FIG. 6C shows a blended image combining the needle frame of FIG. 6A and the composite tissue frame of FIG. 6B. A user viewing the blended image of FIG. 6C can visualize the interventional instrument in relation to the tissue structures in the composite tissue frame.

[0037] In certain embodiments, the ultrasound imaging system can automatically detect whether the interventional instrument is inserted into the anatomical structure from the left or right side. By way of example and not limitation, a user can indicate via user input on the ultrasound imaging system that an interventional instrument is being used along with an anatomical feature of interest. The user can then insert the interventional instrument without specifying the direction in which the interventional instrument is asserted to be valid. The ultrasound imaging system can receive one or more B-mode frames to generate a composite tissue frame, as well as one or more needle frames indicating the position of the interventional instrument.

[0038] In certain embodiments, the ultrasound imaging system can detect the presence of an interventional instrument from only B-mode frames. By way of example and not limitation, the ultrasound imaging system can detect tissue distortion in the B-mode frames consistent with a right-side needle entry into the frame. In certain embodiments, "left" and "right" side entry can be identified from the perspective of the user. In certain embodiments, "left" and "right" side entry may be identified from the perspective of the ultrasound probe. In certain embodiments, the particular term for entry from one direction into tissue or otherwise can be understood to be irrelevant as long as the user is consistently aware of which term refers to which direction. In the examples described in this application, "left" and "right" are used and may be understood to refer to two opposite directions in which needle entry is possible. As another example, the ultrasound imaging system can compare B-mode frames over time and detect movement corresponding to a left-side entry of the interventional instrument. In certain embodiments, based on detection of a left or right entry of the interventional instrument, the ultrasound imaging system can automatically capture one or more needle frames from an angle corresponding to the detected side. As an example and not by way of limitation, if the ultrasound imaging system determines from the B-mode frames that there is a left-side approach of the needle, the ultrasound imaging system can capture three needle frame images corresponding to images of a left-side approach at shallow, medium, and steep angles appropriate for a left-side approach, and then use all three images to identify a linear structure corresponding to the needle.

[0039] In certain embodiments, the ultrasound imaging system can use a set of search needle frames to detect left or right entry of the interventional instrument. By way of example and not limitation, prior to detection of the interventional instrument, the ultrasound imaging system may acquire multiple B-mode frames (e.g., three B-mode frames) to generate a composite tissue frame for display, followed by a left-side medium angle needle frame, followed by a right-side medium angle needle frame. As another example, the ultrasound imaging system can also capture a third needle frame, where the third needle frame cycles from the left-side needle frame and the right-side needle frame. In certain embodiments, the overall frame rate for capturing three B-mode frames followed by three needle frames may be similar to the frame rate when the ultrasound imaging system automatically identifies the best angle for the needle frame, also utilizing three B-mode frames and three needle frames.

[0040] In certain embodiments, the ultrasound imaging system can capture a first set of ultrasound image frames of a target tissue region, the first set corresponding to a plurality of B-mode, color mode, or other suitable image frames for depicting tissue within the target region. By way of example and not limitation, the steering angles for the first set of ultrasound image frames can be −14 degrees, 0 degrees, and +14 degrees, where 0 degrees in this example depicts the angle at which the transmit beam of the ultrasound imaging system is perpendicular to the ultrasound transducer of the ultrasound imaging system. An exemplary coordinate system of positive, negative, and zero steering angles is shown in FIG. 4 and FIG. 5. In certain embodiments, the specific assignment of “positive” steering angles to certain directions and “negative” steering angles to other directions may not be important as long as the system and user can distinguish that they are opposite to each other. The ultrasound imaging system can then capture a second set of ultrasound image frames following the first set. The second set of ultrasound image frames can use steering angles that are steeper than the first set, which may be more suitable for imaging an interventional instrument such as a needle. As an example, the second set of ultrasound image frames may be captured at steering angles of -32 degrees, +24 degrees, and +32 degrees. In certain embodiments, the second set of ultrasound image frames may include at least one frame taken from a first side of zero degrees (i.e., a negative steering angle) and at least one frame taken from an opposite second side (i.e., a positive steering angle). In certain embodiments, the second set of ultrasound image frames may alternate the ratio of steering angles for subsequent frames. By way of example and not limitation, if a previous second set of ultrasound image frames used -32 degrees, +24 degrees, and +32 degrees, the subsequent second set may use -32 degrees, -24 degrees, and +32 degrees. In certain embodiments, an even number of second set frames are captured, and each second set may capture an equal number of negative and positive angle ultrasound image frames.

[0041] In certain embodiments, based on the second set of ultrasound image frames, the ultrasound imaging system identifies whether an interventional instrument, such as a needle, is present in one or more images. In certain embodiments, if none of the second set of ultrasound image frames depict an interventional instrument, the imaging sequence continues as described above. In certain embodiments, if an interventional instrument is detected, the ultrasound imaging system can then identify whether the interventional instrument is depicted with a left approach (e.g., depicted in the negative angle image) or a right approach (e.g., depicted in the positive angle image). Since an image of the interventional instrument can be stronger when the steering angle is perpendicular or nearly perpendicular to the main axis of the interventional instrument (such as the shaft of the needle), the interventional instrument will be depicted much stronger in one direction than in the other direction (where the steering angle is parallel and close to the interventional instrument). In certain embodiments, once a direction is selected, the ultrasound imaging system can generate a composite tissue frame to visualize the interventional instrument by using the first set of ultrasound image frames for tissue and one or more ultrasound image frames taken depicting the interventional instrument. In certain embodiments, the one or more ultrasound image frames may be images already captured in the second set. In certain embodiments, the ultrasound imaging system may identify that additional images are needed to capture the interventional instrument. The ultrasound imaging system may identify a new steering angle for the additional images that may better depict the interventional instrument, as compared to the images of the second set.

[0042] In certain embodiments, upon detecting the interventional instrument and its approach direction, the ultrasound imaging system may adjust the direction and steering angle of the second set of ultrasound image frames for subsequent images. For example, if the previous image set indicates that the interventional instrument is detected with a right-side approach best captured at +32 degrees, then the ultrasound imaging system may capture the subsequent second set only in the positive direction or may always include a +32 degree steering angle to blend with the composite tissue frame depicting the interventional instrument. In this example, the ultrasound imaging system may eliminate the number of captured negative steering angle frames (i.e., consistent with a left-side approach) because the interventional instrument has already been detected with a right-side approach. In certain embodiments, the ultrasound imaging system stops capturing left-side frames for the second set once the right-side approach is identified, and captures only frames of the right-side approach moving forward. In certain embodiments, the ultrasound imaging system continues the normal process to capture the second set of ultrasound image frames until a plurality of the second set confirms the presence and approach direction of the interventional instrument. By way of example and not limitation, if one of the second set of ultrasound image frames indicates a left-side approach, the ultrasound imaging system will still not be able to identify that there is a left-side approach of the interventional instrument until two subsequent sets of ultrasound image frames indicate a left-side approach.

[0043] In certain embodiments, it may be further advantageous for the ultrasound imaging system to automatically detect whether an interventional instrument is present within the field of view of the image of the biological tissue without input from the user indicating that the interventional instrument has been introduced within the field of view. In certain embodiments, the automatic detection of the interventional instrument can be combined with the automatic determination of left or right side entry of the interventional instrument, and the automatic identification of shallow, medium, or steep angles as the optimal needle frame. An advantage of such an embodiment may be that the user does not need to provide any additional input to start imaging the interventional instrument; once the user inserts the interventional instrument, the ultrasound imaging system can detect the instrument, identify the direction and angle, detect linear structures by capturing the needle frame at the appropriate direction and angle, and generate a blended image for display.

[0044] In certain embodiments, automatic detection may be performed by having the ultrasound imaging system always capture at least one needle frame image in addition to the B-mode frames. In certain embodiments, once the ultrasound imaging system has determined that an interventional instrument is present, it may begin capturing additional needle frames to determine the direction of entry and identify the optimal angle for imaging the interventional instrument. In certain embodiments, automatic detection of the interventional instrument may begin by analysis of only the B-mode frames, without the needle frames. By way of example and not limitation, the needle may be detected on an angled B-mode frame (e.g., +15 degrees) that is not of sufficient quality for accurate visualization and display, but is of sufficient quality to determine that an interventional instrument has been inserted and from which direction.

[0045] In certain embodiments, the ultrasound imaging system may utilize one or more classifier algorithms to identify whether an interventional instrument is present in an image of the biological tissue without utilizing needle frame images. In certain embodiments, the classifier algorithm may rely on artificial intelligence, machine learning, neural networks, or any other suitable algorithm to analyze the ultrasound images. By way of example and not limitation, the classifier algorithm may detect tissue distortions in the image of the biological tissue that occur over time due to movement of the interventional instrument affecting the surrounding tissue. As another example and not by way of limitation, the classifier algorithm may be trained to classify images depicting non-periodic temporal changes in the image of the biological tissue as potentially including an interventional instrument. In such an example, the classifier algorithm may identify whether the changes over time in the image of the biological tissue are due to periodic motion (like a heartbeat) and ignore those motions. In certain embodiments, the classifier algorithm may receive positive feedback to detect temporal changes in the structure of the biological tissue corresponding to the insertion of the interventional instrument, and negative feedback to detect temporal changes due to other factors such as blood flow, heartbeat, or muscle movement.

[0046] In certain embodiments, the trained classifier algorithm may be based on a neural network. As is well understood in the art, a neural network includes a number of individual "neuron" algorithms, each of which is trained to detect a particular input and output a value when the input is detected. A neural network may have a number of neurons operating in parallel and a number of layers of neurons that iteratively receive input from a previous layer and provide output to a subsequent layer. By way of example and not limitation, a neural network may be trained to detect left or right entry of an interventional instrument by inputting a number of images, plus information indicating whether an interventional instrument is present, in which direction, and which needle frame angle is appropriate. By training the neurons such that the output based on the input images corresponds to the information known about each image, the neural network may be trained to receive future images and execute the classifier algorithm described herein. In certain embodiments, using a neural network to develop a classifier algorithm allows the training process to be fully automated given a sufficient number of labeled training images. In certain embodiments, given sufficient training images and training feedback, the neural network can detect any other inputs in the ultrasound images that may be relevant to detect the interventional instrument and its direction of entry without requiring explicit input from the user to look for such inputs. In certain embodiments, the use of a trained classifier algorithm may provide an improvement to ultrasound imaging systems in automatically detecting the presence of an interventional instrument, its direction, and in selecting a needle frame angle for imaging the interventional instrument.

[0047] In certain embodiments, the classifier algorithm may be trained only to detect whether an interventional instrument is present. In certain embodiments, the classifier algorithm may also be trained to identify left or right entry of an interventional instrument. By way of example and not limitation, the classifier algorithm may analyze an ultrasound image and return a score ranging from 0 to 1, where 0 indicates "left" and 1 indicates "right". In this example, a score between 0 and 1 may represent the likelihood that the interventional instrument is on the left or right. In certain embodiments, the classifier algorithm may have a first step of identifying whether an interventional instrument is present, and then in a second step, if an interventional instrument is present, then identify a score corresponding to whether there is a left or right entry of the interventional instrument, as described above. In such embodiments, in the first step, if an interventional instrument is not detected, there may be no need to proceed to the second step. In certain embodiments, a threshold score is required to identify that there is a left or right entry. By way of example and not by way of limitation, if 0 corresponds to "left" and 1 corresponds to "right", then a left-side approach may be identified if the score from the classifier algorithm is less than 0.25 and a right-side approach may be identified if the score from the classifier algorithm is greater than 0.75. The threshold may be adjusted automatically by the ultrasound imaging system or may be set by a user. In certain embodiments, identification of a left-side or right-side approach requires a threshold score to be met for multiple frames. By way of example and not by way of limitation, a left-side approach is not identified until three consecutive frames captured by the ultrasound imaging system return a score from the classifier algorithm within the threshold range for a left-side approach. As another example, a right-side approach may be identified if X frames within the Y most recent frames have a score within the threshold range for a right-side approach. In such a case, if 8 of the last 10 frames captured by the ultrasound imaging system show a score corresponding to a right-side approach, then the classifier algorithm may identify that there has been a right-side approach of the interventional instrument.In certain embodiments, the classifier algorithm can identify the entry on a frame-by-frame basis. In certain embodiments, if a score is identified by the classifier algorithm but a threshold score is not met for either left or right entry, the classifier algorithm identifies that an interventional instrument is not actually present. In certain embodiments, if a threshold score is not met, the classifier algorithm can identify that additional images must be captured and analyzed to confirm the presence of an interventional instrument and to identify the direction of entry. In certain embodiments, the classifier algorithm can be further trained to identify which of the shallow, medium, and steep needle frames may be most appropriate for generating a composite image.

[0048] In certain embodiments, a set of training images is used to train the classifier algorithm. By way of example and not limitation, simulated ultrasound images may be generated via MATLAB or other suitable programs such that data in the simulated ultrasound images correspond to images representing an interventional instrument in tissue. The set of training images may depict the interventional instrument at various angles relative to the transducer. In certain embodiments, the set of training images may include simulated needle images superimposed on actual tissue images. In certain embodiments, the set of training images may include in situ images of an interventional instrument placed in organic tissue. In certain embodiments, the set of training images is specific to the type of classifier algorithm being trained. By way of example and not limitation, if a classifier algorithm is trained to detect interventional instruments in angled B-mode images (e.g., −15 degrees and +15 degrees), the set of training images may include labeled B-mode images that may or may not have an interventional instrument. As another example and not by way of limitation, for a classifier algorithm that is trained to detect tissue distortion in a difference frame between a first composite tissue frame and a nearest successive composite tissue frame, the set of training images may include labeled difference frames.

[0049] FIG. 7 illustrates an exemplary system for automatically detecting the presence of an interventional instrument, detecting whether the interventional instrument is entering from the left or right, identifying the appropriate needle frame angle, and generating a blended tissue and instrument image. The ultrasound imaging system captures multiple B-mode frames 710. By way of example and not limitation, three B-mode images at -14 degrees, 0 degrees, and +14 degrees can be captured. As another example, five B-mode images can be captured: two angled from the left, two angled from the right, and one at 0 degrees. In step 720, the ultrasound imaging system can combine the multiple B-mode frames 710 into a composite tissue frame 730. In certain embodiments, the composite tissue frame 730 shows the anatomical tissue structure of interest. In certain embodiments, an interventional instrument may be present while capturing the multiple B-mode images 710.

[0050] In the example of FIG. 7, a first classifier algorithm 740 can receive the composite tissue frame 730 as an input and can identify whether an interventional device is present. By way of example and not limitation, the first classifier algorithm 740 can detect distortions in the composite tissue frame and identify that the distortions correspond to the presence of an interventional device. By way of another example and not limitation, the first classifier algorithm 740 can detect linear structures in the composite tissue frame to identify that an interventional device is present. Upon identifying that an interventional device is present, the first classifier algorithm 740 can output a value indicating whether the interventional device is left or right. In certain embodiments, this output is sent to a voting algorithm 750.

[0051] The composite tissue frame 730 may further be compared to a previous tissue frame captured and displayed by the ultrasound imaging system. By way of example and not limitation, the previous tissue frame may be the closest composite tissue frame in time that precedes the current composite tissue frame 730. Based on this comparison, the ultrasound imaging system may generate a frame difference 742 that represents the temporal change in the composite tissue frame. The frame difference 742 may be input to a second classifier algorithm 744 that is trained to detect an interventional device based on the temporal changes in the tissue. By way of example and not limitation, the distortion of the tissue between the composite tissue frame 730 and the previous tissue frame indicates the presence and location of the interventional device. In certain embodiments, the comparison of the two tissue frames may involve the two frames captured as consecutive frames in time. In certain embodiments, the comparison may use two tissue frames separated by a set amount of time or a set amount of frames. In certain embodiments, the selection of the two tissue frames for comparison may be automatically set by the ultrasound imaging system based on the current image capture settings of the ultrasound imaging system. In certain embodiments, the user can input parameters for differentiation between the two tissue frames being compared.

[0052] In certain embodiments, the second classifier algorithm 744 may detect differences using the B-mode frame 710 rather than the composite tissue frame 730. By way of example and not limitation, the second classifier algorithm 744 may detect changes between a recent B-mode frame acquired at +14 degrees and a prior B-mode frame acquired at +14 degrees and identify whether a change corresponding to an intervening member may be detected. The second classifier algorithm 744 may output a score indicating whether the interventional instrument is to the left or right and may send the output to the voting algorithm 750. In certain embodiments, if the second classifier algorithm 744 does not detect an interventional instrument, it may not send an output score to the voting algorithm 750. In certain embodiments, if the interventional instrument is not detected, the second classifier algorithm 744 may affirmatively indicate to the voting algorithm 750 that it did not detect an interventional instrument.

[0053] The multiple B-mode frames 710 may be sent to a third classifier algorithm 715 . The third classifier algorithm 715 may determine whether linear structures can be detected using the steepest angle B-mode frame. By way of example and not limitation, the plurality of B-mode frames may include frames captured at -12 degrees, -6 degrees, 0 degrees, +6 degrees, and +12 degrees. In such an example, the third classifier algorithm 715 may consider frames captured at -12 degrees and +12 degrees. The third classifier algorithm 715 may determine whether the considered B-mode frames clearly depict an interventional device, as described above. If the third classifier algorithm 715 determines that an interventional device is present, for example, via detection of tissue distortion or detection of linear structures in one or more B-mode frames, it may output a score indicating whether the interventional device is to the left or to the right, and may send that output to the voting algorithm 750. In certain embodiments, the third classifier algorithm 715 is the same as the first classifier algorithm 740, and both are trained to detect tissue distortion or linear structures in one or more frames.

[0054] In certain embodiments, the voting algorithm 750 can receive outputs from the first classifier algorithm 740, the second classifier algorithm 744, and the third classifier algorithm 715, each of which provided an output value including an identification of whether an interventional device is present and from which direction. In certain embodiments, the voting algorithm 750 can determine a weighted average of the outputs from the classifier algorithms to identify the most likely direction and most appropriate angle at which the interventional device was placed. In certain embodiments, the voting algorithm 750 may itself be trained to output certain values ​​for direction and angle based on the particular inputs it received from the classifier algorithms. In certain embodiments, the voting algorithm 750 can calculate an average value corresponding to the direction of entry over a period of time, which can account for any false positives in the images over time. By way of example and not limitation, if the voting algorithm 750 identifies 50 consecutive sets of ultrasound images in which an interventional instrument is inserted from the right, then for two sets of ultrasound images the voting algorithm 750 identifies the interventional instrument as being on the left, and then for a subsequent set of ultrasound images the interventional instrument is detected on the right, the voting algorithm 750 identifies the two identified as being on the left as being in fact false positives, and continues to capture needle frames for right-side entry throughout this period. In certain embodiments, the voting algorithm 750 can identify from the classifier algorithm that no interventional instrument is currently present, and can identify that no needle frames are needed. In certain embodiments, the identification of no interventional instrument is made only if a sufficient number of consecutive sets of ultrasound images do not show the interventional instrument, thereby smoothing out false negatives.

[0055] In step 755, the voting algorithm 750 may adaptively control the signal path control (SPCTL) of the ultrasound imaging system. The SPCTL may be used to update the frame sequencer of the ultrasound imaging system to capture a left or right needle frame 760 at an angle specifically suited for the interventional instrument. In certain embodiments, based on the output of the voting algorithm 750, only one needle frame 760 is needed to provide a blended image. In certain embodiments, based on the output of the voting algorithm 750, the ultrasound imaging system may determine that a needle frame is not needed to visualize the interventional instrument. By way of example and not limitation, the voting algorithm 750 may determine that the steep B-mode frame used for the composite tissue frame (and the third classifier algorithm) is sufficient for detection of linear structures.

[0056] In step 765, based on the needle frame 760 captured for visualization of the interventional device, the ultrasound imaging system can identify whether a linear structure exists in the needle frame 760. The ultrasound imaging system can subsequently generate an enhanced linear structure 770, which is registered to the tissue frame. In step 780, the mixed tissue frame 730 and the enhanced linear structure 770 are combined into a mixed image 790, which may then be displayed.

[0057] Particular embodiments may repeat one or more steps disclosed in Figure 7 if necessary. Although this disclosure describes and illustrates certain steps of the method of Figure 7 as occurring in a particular order, this disclosure contemplates any suitable steps of the method of Figure 7 occurring in any suitable order. Furthermore, although this disclosure describes and illustrates particular components, devices, or systems that perform certain steps of the method of Figure 7, this disclosure contemplates any suitable combination of any suitable components, apparatus, or systems that perform any suitable steps of the method of Figure 7.

[0058] In certain embodiments, the present invention utilizing always-on detection of interventional instruments and automatic left / right entry detection of interventional instruments can provide an advantage over previous needle entry detection systems by improving the frame rate at which a mixed image can be generated. By way of example and not limitation, by reducing the number of needle frames that must be captured during visualization of the instrument to one or zero frames (from three or more frames), less time is required to capture a needle frame for use in the mixed frames, effectively increasing the frame rate of ultrasound imaging. In certain embodiments, the frame rate of the mixed image can be improved in direct proportion to the amount of reduction in needle frames required. In certain embodiments, another advantage of the present invention may be an improved user experience in using an ultrasound imaging system by not needing to divert the user's attention to specifying (1) that an interventional instrument will be used and (2) the direction in which the instrument is being inserted.

[0059] The subject matter and operations described herein may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, or in any combination or combinations thereof, including the structures disclosed herein and their structural equivalents.Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by or for controlling the operation of a data processing apparatus.

[0060] A computer storage medium can be or can be included in a computer-readable recording device, a computer-readable recording substrate, a random or serial access memory array or device, or a combination of one or more thereof. Furthermore, while a computer storage medium is not a propagating signal, a computer storage medium can be the source or destination of computer program instructions encoded in an artificially generated propagating signal. A computer storage medium can also be or can be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0061] The term "processor" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a combination of the foregoing. The apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0062] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. A program may be stored in one or more files dedicated to the program, or in multiple coordinated files (e.g., files storing one or more modules, subprograms, or portions of code), among other files that hold other programs or data (e.g., one or more scripts stored in a markup language document). A computer program can be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0063] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and an apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0064] Processors suitable for executing computer programs can include, by way of example and not limitation, both general purpose and special purpose microprocessors. Devices suitable for storing computer program instructions and data can include all forms of non-volatile memory, media, and memory devices, including, by way of example and not limitation, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and disks such as CD-ROM and DVD-ROM. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0065] From the foregoing, it will be appreciated that, although specific embodiments of the invention have been described herein for purposes of illustration, various modifications can be made without deviating from the scope of the invention. Accordingly, the invention is not to be limited except as by the appended claims.

Claims

1. 1. An ultrasound imaging system including an image processor, the ultrasound imaging system comprising: Capturing a first set of ultrasound image frames of the target area; capturing a second set of ultrasound image frames of the target area; the second set of ultrasound image frames being captured by the ultrasound imaging system subsequent to the first set of ultrasound image frames; At least one of the second set of ultrasound image frames is acquired on a first side having a positive steering angle, the positive steering angle being identified relative to a zero angle at which a transmit beam of the ultrasound imaging system is perpendicular to an ultrasound transducer of the ultrasound imaging system; and At least another one of the second set of ultrasound image frames is acquired from a second side having a negative steering angle relative to the zero angle. capturing a second set of ultrasound image frames of the target area; determining whether an interventional device is present within the target area based on at least the second set of ultrasound image frames; In response to determining that the interventional instrument is present, determining whether the interventional instrument is depicted by image frames from a steering angle of the first side or the second side; and in response to identifying the side, generating a composite tissue frame depicting tissue of the target area and the interventional device based on at least the first set of ultrasound image frames and a third set of ultrasound image frames acquired from a steering angle at the identified side. It is configured as follows: the third set of ultrasound image frames includes one or more ultrasound image frames from the second set of ultrasound image frames. Ultrasound imaging systems.

2. the third set of ultrasound image frames is additionally captured by the ultrasound imaging system in response to identifying the side. The ultrasound imaging system of claim 1 .

3. the ultrasound imaging system is further configured, in response to identifying the side, to only capture, for a second set of subsequent ultrasound image frames, ultrasound image frames taken at steering angles at the identified side. The ultrasound imaging system of claim 1 .

4. the ultrasound imaging system is further configured, in response to identifying the side, to select a steering angle for depicting the interventional instrument from a plurality of steering angles at the identified side, wherein selecting the steering angle is based on detecting that the selected steering angle is approximately perpendicular to an orientation of the interventional instrument. The ultrasound imaging system of claim 1 .

5. The ultrasound imaging system is further configured, in response to selecting the steering angle, to use the selected steering angle to capture a second set of a plurality of subsequent ultrasound image frames. The ultrasound imaging system of claim 4.

6. the second set of ultrasound image frames are captured using a second steering angle that is steeper than a first steering angle used to capture the first set of ultrasound image frames. The ultrasound imaging system of claim 1 .

7. the ultrasound imaging system is further configured to determine a direction of entry of the interventional instrument into the tissue based on the determined side. The ultrasound imaging system of claim 1 .

8. the ultrasound imaging system is further configured to adjust a direction and a steering angle to capture a second set of subsequent ultrasound image frames in response to identifying the side. The ultrasound imaging system of claim 1 .

9. The first set of ultrasound image frames comprises: at least one ultrasound image frame captured at a steering angle of 0 degrees; and at least one additional ultrasound image frame captured at a steering angle between −15 degrees and +15 degrees; Including, The ultrasound imaging system of claim 1 .

10. determining whether the interventional device is present based on one or more classifier algorithms trained to detect interventional devices; the one or more classifier algorithms comprising a classification algorithm trained to classify images depicting non-periodic temporal changes in the second set of ultrasound image frames as including the interventional device. The ultrasound imaging system of claim 1 .

11. the one or more classifier algorithms comprising a first classifier algorithm trained to identify whether an interventional device is present based on detection of distorting tissue in one or more of the second set of ultrasound image frames. The ultrasound imaging system of claim 10.

12. Multiple outputs from one or more of the classifier algorithms are combined to identify the likelihood that an interventional device is present. The ultrasound imaging system of claim 10.

13. the combination of the plurality of outputs comprises a weighted average of the plurality of outputs.

13. The ultrasound imaging system of claim 12.

Images