Analyzing ultrasound image feeds
By detecting and highlighting the target tissue in ultrasound image feed, the problem of coupling image acquisition and interpretation in breast US examination is solved, and the continuous tracking and diagnostic efficiency of the target tissue is improved.
Patent Information
- Application Number
- CN202380036357.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-07
- Filing Date
- 2023-04-25
- Publication Date
- 2024-12-06
AI Technical Summary
The existing ultrasound image analysis workflow is coupled with user interpretation in breast US examination, and cannot be separated, resulting in inability to apply to certain types of examinations. The output marking method of ML models is complex, which may lead to distraction or insufficient information. .
By detecting the target tissue in the ultrasound image feed, if its visibility is above the threshold, a message is sent to the display to highlight the target tissue and keep it highlighted in subsequent images, ensuring that the user can reliably track the detected target tissue.
Improved analysis tools during ultrasound examination are provided to ensure that users can continuously track target tissue in the image sequence, avoid flickering of test result markers, and improve the reliability and efficiency of diagnosis.
Smart Images

Figure CN119096305A_ABST
Abstract
Description
Technical Field
[0001] The disclosure herein relates to analyzing ultrasound image feeds. Specifically, the ultrasound image feeds are not a time series of contrast-enhanced ultrasound images. Some embodiments herein relate to causing a display to highlight target tissue in an ultrasound image feed. Background Art
[0002] There are various tools to assist clinicians (e.g., radiologic technicians) in analyzing and interpreting ultrasound (US) image data to identify abnormalities. US image data can be a sequence of ultrasound images (also referred to as ultrasound frames). It can be a sequence of ultrasound images acquired when an ultrasound probe and / or imaging plane or volume moves relative to a region of interest, or it can be a sequence of ultrasound images of a moving object (e.g., a heart). Some tools use, for example, machine learning (ML) models to assist the diagnostic process. For example, an ML model can be used to identify a specific tissue (e.g., a lesion).
[0003] ML models can be used to identify data that may reveal abnormalities during a clinician's interpretation of a patient's radiological images. For example, ML models can be used for lung nodule detection in computed tomography (CT) images and lesion identification in US images of breast tissue, among other tasks. Such models may be referred to as computer-aided detection (CADe) models. For example, see Kozegar et al., “Computer aided detection in automated 3-D breast ultrasound images: a survey” (Artificial Intelligence Review 53, pp. 1919-1941, 2020).
[0004] While such ML models are useful, there is a complex relationship between the output of such ML models and clinicians, and as such, clinical performance may depend on a variety of factors that should be considered in any study design, including: the timing of the use of the ML model in the interpretation process (e.g., simultaneous reading or secondary reading); the physical characteristics of the markers used to highlight the output of the ML model on the US image (e.g., size and shape), the type of border used to indicate the bounding box (e.g., solid line, dashed line, circle, etc. outline), and the proximity of such markers to the detected tissue or abnormality. In addition, the user's knowledge of the tissue type that the ML model is designed to label can affect the overall clinical performance. The number of markers provided by the ML model can also affect the manner and efficacy of using the ML tool (e.g., too many markers may confuse or distract the user, while too few markers may not convey enough information for the user to fully understand the output of the ML model).
[0005] Therefore, the above points need to be carefully considered when designing and integrating ML models and exporting them into workflows to ensure that clinicians can fully benefit from the assistance of ML models. Summary of the invention
[0006] Many existing workflows for medical image analysis that incorporate ML models (e.g., ML models for lung nodule detection in CT images) are designed in a post-acquisition, assisted-reading manner. This means that the images are first acquired and analyzed by the ML model, and then the detection results are displayed to the user (e.g., radiologist) during the interpretation phase.
[0007] However, this workflow is not suitable for certain types of examinations (eg, breast US examinations, where image acquisition and user interpretation are inherently coupled and cannot be separated). Our approach aims to provide an improved workflow and analysis tool for US image feeds.
[0008] According to a first aspect of the present invention, a computer-implemented method for analyzing an ultrasound image feed of a region of interest as part of an ultrasound US examination is provided. The method comprises: detecting a target tissue in a first image of the US image feed; if the target tissue has a visibility above a first visibility threshold, sending a message to a display to cause the display to highlight the target tissue in the first image on the display; and causing the display to highlight the target tissue in each successive US image in the US image feed after the first image in which the target tissue is visible. The US image feed is not a temporal sequence of contrast-enhanced ultrasound images. It can include a sequence of ultrasound images of a moving object (e.g., a heart). It can include a sequence of ultrasound images acquired when an ultrasound probe and / or an imaging plane or volume moves relative to a region of interest. For example, a sequence of ultrasound images, such as those acquired during a sweep scan, is acquired when the ultrasound probe moves relative to the region of interest. That is, in some embodiments, the US image feed includes a sequence of ultrasound images acquired when the relative spatial position between the field of view of the ultrasound image and the region of interest changes. The field of view of two-dimensional ultrasound is generally referred to as an imaging plane, while the field of view of three-dimensional ultrasound is generally referred to as an imaging volume. In some embodiments, the region of interest includes breast tissue and the US image feed includes images of the breast tissue as part of an examination for breast lesions.
[0009] According to a second aspect, there is provided an apparatus for analyzing an ultrasound image feed as part of an ultrasound (US) examination. The system comprises: a memory comprising instruction data representing a set of instructions; and a processor configured to communicate with the memory and execute the set of instructions. The set of instructions, when executed by the processor, causes the processor to: detect a target tissue in a first image of the US image feed; if the target tissue has a visibility above a first visibility threshold, send a message to a display to cause the display to highlight the target tissue in the first image on the display; and send another message to cause the display to highlight the target tissue in each successive US image in the US image feed, after the first image, in which the target tissue is visible.
[0010] According to a third aspect, there is provided an ultrasound (US) imaging system comprising the apparatus of the second aspect and a display for displaying the ultrasound image feed. The message and the further message cause the display to highlight the target tissue in the first image and the target tissue in the subsequent US images on the display.
[0011] According to a fourth aspect, there is provided a computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that when run by a suitable computer or processor, the computer or processor is caused to perform the method of the first aspect.
[0012] Current methods of performing detection during a US examination are typically performed on an image-by-image (e.g., frame-by-frame) basis. As a result, if the visibility of the target tissue varies between different US images or views in the US examination, this can cause any markers used to highlight detected target tissue to flash on and off depending on whether the target tissue in any particular image in the image feed appears just above or just below the detection threshold.
[0013] In the above-described embodiments, once the visibility rises above the target threshold, the target tissue is highlighted and is also highlighted in all subsequent sequential images thereafter. This enables the user to reliably track the detected target tissue between different images in a US image feed (where the target tissue may have a variable visibility level). Thus, an improved method and system for assisting a user in analyzing / verifying automatically detected target tissue during live ultrasound scanning is provided.
[0014] These and other aspects will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Example embodiments will now be described, by way of example only, with reference to the following drawings, in which:
[0016] Figure 1 shows an apparatus according to some embodiments herein;
[0017] Figure 2 A US system according to some embodiments herein is shown;
[0018] Figure 3 Methods according to some embodiments of the present invention are shown;
[0019] Figure 4 illustrates a method according to some embodiments herein;
[0020] Figure 5 illustrates a user interface according to some embodiments herein; and
[0021] Figure 6 Another user interface according to some embodiments herein is illustrated. DETAILED DESCRIPTION
[0022] As described above, one way to incorporate ML model detection of target tissue into the US workflow is to display the detection results on each acquired image (e.g., frame), for example, by attaching a marker (e.g., a bounding box around the target tissue / abnormality) on each acquired image, but this approach has some problems.
[0023] In some cases, a marker attached to the acquired image may influence the diagnosis of the user (who may be a radiologist or other clinician, as described above) if certain features of the abnormality are obscured by the marker. If the abnormality appears in a short period of time (e.g., a small number of images in a US image feed), the user may ignore the marker. In addition, in many workflows incorporating ML models, a predefined threshold will be used to filter the detection results, and only those detection results with a prediction confidence exceeding the predefined threshold will be pushed to the user, however, the prediction confidence of the same abnormality across successive images will vary greatly, which can lead to the following situation: for a single abnormality throughout the entire process of the US examination, the detection result marker will appear (possibly multiple times) and disappear afterwards. This "flash" marker is suboptimal and distracts the user.
[0024] In short, this paper proposes a method according to which detection results of target tissue (e.g., abnormality) above a predefined threshold in a first US image are tracked between multiple images in a US image feed, and all US images subsequent to the first US image in which the target tissue is visible (even if below the predefined threshold) are highlighted to the user. This provides detection continuity between different detection views of the target tissue and also prevents the aforementioned problem of detection result marks appearing and disappearing in a flickering manner.
[0025] In more detail, we now turn to Figure 1 In some embodiments, there is an apparatus 100 for analyzing ultrasound US image feeds according to some embodiments herein. Typically, the apparatus may form part of a computer apparatus or system (e.g., a laptop, desktop computer or other computing device). In some embodiments, the apparatus 100 may form part of a distributed computing arrangement or cloud.
[0026] The apparatus includes: a memory 104 including instruction data representing a set of instructions; and a processor 102 (e.g., a processing circuit or a logic unit) configured to communicate with the memory and execute the set of instructions. Generally, the set of instructions, when executed by the processor, can cause the processor to perform any embodiment of the method 300 described below.
[0027] Embodiments of the apparatus 100 may be used to analyze an ultrasound US image feed. More specifically, the set of instructions, when executed by the processor, causes the processor to: detect a target tissue in a first image of the US image feed; if the target tissue has a visibility above a first visibility threshold, send a message to a display to cause the display to highlight the target tissue in the first image on the display; and send an additional message to cause the display to highlight the target tissue in each successive US image in the US image feed, after the first image, in which the target tissue is visible.
[0028] Processor 102 can include one or more processors, processing units, multi-core processors or modules, which are configured or programmed to control device 100 in a manner described herein. In a specific embodiment, processor 102 can include multiple software modules and / or hardware modules, which are all configured to perform or are used to perform individual steps or multiple steps of the method described herein. Processor 102 can include one or more processors, processing units, multi-core processors and / or modules, which are configured or programmed to control device 100 in a manner described herein. In some embodiments, for example, processor 102 may include multiple (e.g., interoperable) processors, processing units, multi-core processors and / or modules configured for distributed processing. Those skilled in the art will appreciate that such processors, processing units, multi-core processors and / or modules can be located in different locations and can perform different steps of the method described herein and / or different parts of a single step.
[0029] The memory 104 is configured to store program code that can be executed by the processor 102 to perform the methods described herein. Alternatively or additionally, one or more memories 104 can be external to the device 100 (i.e., separate from or away from the device 100). For example, one or more memories 104 can be part of another device. The memory 104 can be used to store US image feeds, information related to detected target tissues, and / or any other information or data received, calculated, or determined by the processor 102 of the device 100 or from any interface, memory, or device external to the device 100. The processor 102 can be configured to control the memory 104 to store US image feeds, information related to detected target tissues, and / or any other information or data.
[0030] In some embodiments, the memory 104 may include multiple sub-memories, each of which may store one instruction data. For example, at least one sub-memory may store instruction data representing at least one instruction in the set of instructions, and at least one other sub-memory may store instruction data representing at least one other instruction in the set of instructions.
[0031] will realize that Figure 1Only the components required for illustrating this aspect of the present disclosure are shown, and in actual embodiments, the device 100 may include additional components for the components shown. For example, the device 100 may also include a display. The display may include, for example, a computer screen (e.g., a liquid crystal display (LCD)) and / or a screen on a mobile phone or tablet computer. The device may also include a user input device (e.g., a keyboard, a mouse, or other input devices that enable a user to interact with the device), such as to provide an input to be used in the method described herein. The device 100 may include a battery or other power source for powering the device 100 or a module for connecting the device 100 to a mains power supply.
[0032] In some embodiments, the device is incorporated into a US imaging system. For example, an ultrasound US imaging system may include the device 100 and a display to display an ultrasound image feed and highlight results.
[0033] The US imaging system may also include other components (e.g., those associated with obtaining and processing US image data). Figure 2 An example US imaging system 200 is shown in FIG. The US system 200 includes an array transducer probe 4 having a transducer array 6 for transmitting ultrasound waves and receiving echo information. The transducer array 6 may include CMUT transducers; piezoelectric transducers (formed from materials such as PZT or PVDF); or any other suitable transducer technology. In this example, the transducer array 6 is a two-dimensional array of transducers 8 capable of scanning a 2D plane or a three-dimensional volume of a region of interest. In another example, the transducer array may be a 1D array.
[0034] The transducer array 6 may be coupled to a microbeamformer 12 that controls the reception of signals by the transducer elements. The microbeamformer is capable of at least partially beamforming signals received by sub-arrays (often referred to as "groups" or "patches") of transducers as described in U.S. Pat. Nos. 5,997,479 (Savord et al.), 6,013,032 (Savord), and 6,623,432 (Powers et al.).
[0035] In an alternative embodiment, instead of the microbeamformer 12, the transducer array may be provided by a main system beamformer ( Figure 2 ) directly operated.
[0036] The system 200 may also include a transmit / receive (T / R) switch 16 to which the microbeamformer 12 can be coupled and which switches the array between a transmit mode and a receive mode. Transmit operations to transmit ultrasound beams from the transducer array 6 are directed by a controller 18 coupled to the microbeamformer via the T / R switch 16 and a main transmit beamformer (not shown) that can receive input from user operation of a user interface or control panel 38. The controller 18 can include transmit circuitry arranged to drive the transducer elements of the array 6 (directly or via the microbeamformer) during the transmit mode.
[0037] Note that in an alternative embodiment, instead of the microbeamformer 12, the transducer array is operated directly by the main system beamformer, and the T / R switch 16 can protect the main beamformer 20 from high energy transmit signals.
[0038] In a typical line-by-line imaging sequence, the beamforming system within the probe may operate as follows. During transmit, the beamformer (which may be a microbeamformer or a main system beamformer, depending on the implementation) activates a transducer array or a subaperture of a transducer array. A subaperture may be a one-dimensional transducer row or a two-dimensional transducer patch within a larger array. In transmit mode, the focusing and steering of the ultrasound beam generated by the array or a subaperture of the array is controlled as described below.
[0039] When a backscattered echo signal is received from the object, the received signal is subjected to receive beamforming (described below) in order to align the received signal and, in the case of a subaperture, then shift the subaperture, for example, by one transducer element. The shifted subaperture is then activated, and the process is repeated until all transducer elements of the transducer array have been activated.
[0040] For each line (or sub-aperture), the total received signal for the associated line used to form the final ultrasound image will be the sum of the voltage signals measured by the transducer elements of the given sub-aperture during the receive period. After the beamforming process below, the resulting line signal is typically referred to as radio frequency (RF) data. Each line signal (RF data set) generated by the various sub-apertures is then subjected to additional processing to generate the lines of the final ultrasound image. The change in the amplitude of the line signal over time will contribute to the change in the brightness of the ultrasound image with depth, where high amplitude peaks will correspond to bright pixels (or sets of pixels) in the final image. Peaks that appear near the beginning of the line signal will represent echoes from shallow structures, while peaks that gradually appear later in the line signal represent echoes from structures at increasing depths within the object.
[0041] One of the functions controlled by the transducer controller 18 is the direction in which the beam is steered and focused. The beam can be steered straight ahead (orthogonal to the transducer array) from the transducer array, or at different angles for a wider field of view. The steering and focusing of the transmit beam can be controlled based on the transducer element actuation time.
[0042] In general ultrasound data acquisition, two methods are distinguished: plane wave imaging and "beam steering" imaging. The two methods are distinguished by the presence of beam forming in the transmit mode ("beam steering" imaging) and / or the presence of beam forming in the receive mode (plane wave imaging and "beam steering" imaging).
[0043] First, let's look at the focusing function. By activating all transducer elements simultaneously, the transducer array generates a plane wave that diverges as it travels through the object. In this case, the beam of ultrasound waves remains unfocused. By introducing a position-dependent time delay into the activation of the transducer, the wavefront of the beam can be made to converge at a desired point, known as the focal zone. The focal zone is defined as a point where the lateral beam width is less than half the transmit beam width. In this way, the lateral resolution of the final ultrasound image is improved.
[0044] For example, if the time delay causes the transducer elements to be activated in series, starting with the outermost elements and finishing at the center element(s) of the transducer array, then the focal zone will be formed at a given distance away from the probe in line with the center element(s). The distance of the focal zone from the probe will vary depending on the time delay between each subsequent round of transducer element activation. After the beam passes through the focal zone, the beam will begin to diverge, forming a far-field imaging region. It should be noted that for a focal zone positioned close to the transducer array, the ultrasound beam will diverge rapidly in the far field, resulting in beam width artifacts in the final image. Typically, the near field located between the transducer array and the focal zone shows little detail due to the large overlap in the ultrasound beam. Therefore, changing the position of the focal zone can significantly change the quality of the final image.
[0045] It should be noted that in transmit mode, unless the ultrasound image is divided into multiple focal zones (each of which may have a different transmit focus), only one focus may be defined.
[0046] Additionally, upon receiving echo signals from within the object, the inverse of the above process may be performed to perform receive focusing. In other words, the incoming signal may be received by the transducer elements and subjected to an electronic time delay before being passed into the system for signal processing. The simplest example of this is known as delay-and-sum beamforming. The receive focus of the transducer array may be adjusted dynamically based on time.
[0047] Turning now to the function of beam steering, by properly applying time delays to the transducer elements, a desired angle can be imposed on the ultrasound beam as it leaves the transducer array. For example, by activating a transducer on a first side of the transducer array, followed by activating the remaining transducers in a sequence ending at the opposite side of the array, the wavefront of the beam will be tilted toward the second side. The magnitude of the steering angle relative to the normal to the transducer array depends on the magnitude of the time delay between subsequent transducer element activations.
[0048] Additionally, the steered beam can be focused, where the total time delay applied to each transducer element is the sum of the focusing time delay and the steering time delay. In this case, the transducer array is called a phased array.
[0049] In the case of CMUT transducers (which require a DC bias voltage for their activation), the transducer controller 18 can be coupled to control a DC bias control 45 for the transducer array. The DC bias control 45 sets the DC bias voltage(s) to be applied to the CMUT transducer elements.
[0050] For each transducer element of the transducer array, an analog ultrasound signal (commonly referred to as channel data) enters the system through a receive channel. In the receive channel, a partially beamformed signal is generated by the microbeamformer 12 based on the channel data and then passed to the main receive beamformer 20, where the partially beamformed signals from the individual transducer tiles are combined into a fully beamformed signal (referred to as radio frequency (RF) data). The beamforming performed at each stage can be performed as described above, or additional functions can be included. For example, the main beamformer 20 can have 128 channels, each of which receives a partially beamformed signal from a tile of dozens or hundreds of transducer elements. In this way, the signals received by thousands of transducers of the transducer array can effectively contribute to a single beamformed signal.
[0051] The beamformed received signals are coupled to a signal processor 22. The signal processor 22 can process the received echo signals in various ways, such as: bandpass filtering; decimation; separation of the I component from the Q component; and harmonic signal separation, which is used to separate linear signals from nonlinear signals so that nonlinear (higher harmonics of the fundamental frequency) echo signals returning from tissue and microbubbles can be identified. The signal processor can also perform additional signal enhancement, such as speckle reduction, signal compounding, and noise removal. The bandpass filter in the signal processor can be a tracking filter whose passband slides from a higher frequency band to a lower frequency band as the echo signal is received from an increasing depth, thereby rejecting noise at higher frequencies (which come from greater depths where there is generally no anatomical information).
[0052] The beamformer for transmission and the beamformer for reception are implemented in different hardware and can have different functions. Of course, the receiver beamformer is designed to take into account the characteristics of the transmit beamformer. Figure 2 In the illustration, for simplicity, only the receiver beamformer 12, the receiver beamformer 20 are shown. In a complete system, there would also be a transmit chain with a transmit microbeamformer and a main transmit beamformer.
[0053] The function of the microbeamformer 12 is to provide initial combining of the signals in order to reduce the number of analog signal paths. This is typically performed in the analog domain.
[0054] Final beamforming is done in the main beamformer 20 and is typically done after digitization.
[0055] The transmit and receive channels use the same transducer array 6 with fixed frequency bands. However, the bandwidth occupied by the transmit pulses can vary depending on the transmit beamforming used. The receive channel can capture the entire transducer bandwidth (this is the classical approach), or by using bandpass processing, the receive channel can extract only the bandwidth containing the desired information (e.g., harmonics of the main harmonic).
[0056] The RF signal can then be coupled to a B-mode (i.e., brightness mode or 2D imaging mode) processor 26 and a Doppler processor 28. The B-mode processor 26 performs amplitude detection on the received ultrasound signal to image structures in the body (e.g., organ tissue and blood vessels). In the case of line-by-line imaging, each line (beam) is represented by an associated RF signal, whose amplitude is used to generate a brightness value to be assigned to a pixel in the B-mode image. The exact position of the pixel within the image is determined by the position along the associated amplitude measurement of the RF signal and the number of lines (beams) of the RF signal. B-mode images of such structures can be formed in a harmonic image mode or a fundamental image mode or a combination of the two, as described in U.S. Pat. No. 6,283,919 (Roundhill et al.) and U.S. Pat. No. 6,458,083 (Jago et al.). The Doppler processor 28 processes the temporally different signals caused by tissue movement and blood flow to detect moving matter (e.g., the flow of blood cells in the image field). The Doppler processor 28 typically includes a wall filter having parameters set to pass or reject echoes returning from selected types of material within the body.
[0057] The structural and motion signals generated by the B-mode processor and the Doppler processor are coupled to the scan converter 32 and the multi-plane reformatter 44. The scan converter 32 arranges the echo signals in the spatial relationship from which they are received in the desired image format. In other words, the scan converter is used to convert the RF data from a cylindrical coordinate system to a Cartesian coordinate system suitable for displaying an ultrasound image on the image display 40. In the case of B-mode imaging, the brightness of a pixel at a given coordinate is proportional to the amplitude of the RF signal received from that position. For example, the scan converter can arrange the echo signals into a two-dimensional (2D) sector format or a pyramid three-dimensional (3D) image. The scan converter can overlay the B-mode structural image with a color corresponding to the motion at a point in the image field, where the Doppler estimated velocity is used to generate a given color. The combined B-mode structural image and color Doppler image can depict tissue motion and blood flow within the structural image field. The multi-plane reformatter converts the echoes received from points in a common plane in a volumetric region of the body into an ultrasound image of the plane, as described in U.S. Pat. No. 6,443,896 (Detmer). The volume renderer 42 converts the echo signals of the 3D data set into a projected 3D image as viewed from a given reference point, as described in US Pat. No. 6,530,885 (Entrekin et al.).
[0058] The 2D or 3D images are coupled from the scan converter 32, the multi-planar reformatter 44, and the volume renderer 42 to the image processor 30 for further enhancement, buffering, and temporary storage for display on the image display 40. The imaging processor may be adapted to remove certain imaging artifacts (e.g., acoustic shadowing (e.g., caused by strong attenuators or refraction), post-enhancement (e.g., caused by weak attenuators), reverberation artifacts (e.g., where multiple highly reflective tissue interfaces are located in close proximity), etc.) from the final ultrasound image. In addition, the image processor may be adapted to process certain speckle reduction functions to improve the contrast of the final ultrasound image.
[0059] In addition to being used for imaging, the blood flow values produced by the Doppler processor 28 and the tissue structure information produced by the B-mode processor 26 are also coupled to a quantification processor 34. The quantification processor can be used to make measurements in the image. The quantification processor can receive input from a user control panel 38.
[0060] Output data from the quantization processor is coupled to a graphics processor 36 for reproducing measurement graphs and values together with the image on the display 40, and for audio output from the display device 40. The graphics processor 36 can also generate graphic overlays for display with the ultrasound image. These graphic overlays can contain standard identification information (e.g., patient name), date and time of the image, imaging parameters, etc. For these purposes, the graphics processor receives input (e.g., patient name) from a user interface 38. The user interface is also coupled to the transmit controller 18 to control the generation operation of the ultrasound signal generated from the transducer array 6, and thus control the image produced by the transducer array and the ultrasound system. The transmit control function of the controller 18 is only one of the functions performed. The controller 18 also takes into account the operating mode (given by the user) in the receiver analog-to-digital converter and the corresponding required transmitter configuration and bandpass configuration. The controller 18 can be a state machine with fixed states.
[0061] The user interface is also coupled to the multi-planar reformatter 44 for selecting and controlling the planes of a plurality of multi-planar reformatted (MPR) images that may be used to perform quantitative measurements in the image fields of the MPR images.
[0062] will realize that Figure 2 The US image system illustrated in is merely an example, and the US image system may include components different from those described above.
[0063] Now go to Figure 3 , there is a computer-implemented method 300 for analyzing a US image feed. Embodiments of the method 300 may be performed, for example, by an apparatus such as the apparatus 100 or the US imaging system 200 described above.
[0064] The method 300 may be performed to assist a user (eg, a clinician or radiologist) in performing a US examination on a subject to identify target tissue in the subject.
[0065] Briefly, in a first step 302, the method 300 comprises: detecting a target tissue in a first image of a US image feed. In a second step 304, the method 300 comprises: sending a message to a display to cause the display to highlight the target tissue in the first image on the display if the target tissue has a visibility above a first visibility threshold. In a third step, the method 300 comprises: causing the display to highlight the target tissue in each successive US image in the US image feed, after the first image, in which the target tissue is visible.
[0066] In more detail, the US image feed includes a plurality of US images (or image frames). The US image feed may be a real-time (or near real-time) feed. In other words, the US image feed may be a live US image feed, for example, a feed in the process of being obtained as part of a US examination. Figure 2 A US system such as the illustrated system 200 is used to obtain a US image feed.
[0067] In step 302, the method includes detecting a target tissue in a first image of a US image feed. As used herein, a "target tissue" or "target tissue type" may be any tissue of interest, such as an abnormal tissue or an abnormality (e.g., a lesion or a particular type of lesion). In other embodiments, the target tissue may be normal tissue. Typically, a US image feed is obtained to determine whether the target tissue is present in a subject being examined by ultrasound.
[0068] Any known technique (eg, image processing technique) may be used to detect the target tissue. A computer-aided detection (CADe) model may be used to detect the target tissue.
[0069] In some embodiments, a machine learning technique may be used to detect the target tissue in step 302. For example, the step of detecting the target tissue in the first image of the US image feed may be performed using a machine learning model trained to detect the target tissue in the US image. For example, the machine learning model is trained using training data including training examples, each training example including: an example US image and (one or more) corresponding correct detection results (e.g., example input and example ground truth output) of the target tissue in the example US image.
[0070] The skilled person will be familiar with machine learning (ML) models that can be trained to detect features such as a particular type of target tissue in a US image. For example, a convolutional neural network can be trained to detect target tissue in a US image using training data including example US image(s) and example ground truth (e.g., "correct") annotations of whether the corresponding example US image(s) contain the target tissue type.
[0071] As an example, the You Only Look Once (YOLO) network described by Redmon et al. in an article titled “You Only Look Once: Unified Real-Time Object Detection” (2016) can be trained to detect target tissue in US images in real time within a US image feed.
[0072] As another example, the U-net network described by Olaf Ronnetberger et al. in the article “U-Net: Convolutional Networks for Biomedical Image Segmentation” can be trained to segment target tissue from a US image feed.
[0073] However, those skilled in the art will appreciate that these are merely examples and that there are many ways (e.g., using techniques such as radial gradient index (RGI) filtering, multifractal filtering, rule-based region ranking and / or deformable part models) in which target tissue may be detected in US images and US image feeds.
[0074] The output of step 302 may be an indication of one or more detection results for the target tissue (accompanied by a corresponding visibility for each detection). In this sense, the visibility may be, for example, a brightness level or score associated with the detection result, a confidence level associated with the detection result, or any other metric that may be used as an indication of whether the detection result of the target tissue is a true detection result (or a false positive detection result).
[0075] In embodiments where machine learning is used to perform step 302 , the visibility may be based on a confidence score (eg, a probability score) output by the model to indicate the probability that the output of the model is correct.
[0076] The detection technique may be performed on each US image in the US image feed (eg, sequentially).
[0077] In step 304, if the target tissue (e.g., the target tissue detected in step 302) has a visibility above a first visibility threshold, the method 300 includes sending a message to the display to cause the display to highlight the target tissue in the first image on the display. In other words, the first US image is the first US image in the sequence that contains a detection result having a visibility above the first visibility threshold, and the detection result acts as a trigger for highlighting the target tissue detected in the first US image.
[0078] The first visibility threshold may be, for example, a threshold brightness level. In embodiments where machine learning is used to perform the detection in step 302, the first visibility threshold may be a threshold confidence level in the detection result of the target tissue. In other words, if (or in response to) the target tissue is detected with a confidence level above the threshold confidence level, step 304 may be performed.
[0079] The target tissue may be highlighted on the display in any manner. For example, using a bounding box or circle, a text string or alert, a symbol, using a color to highlight (or fill) the area including the detected target tissue, or using any other marking that may draw attention to the detected target tissue. In some embodiments where the US image feed is a live or real-time image feed, the message may also cause the display to display an alert that a live detection has occurred.
[0080] In step 306 , the method then includes causing the display to highlight the target tissue in each successive US image in the US image feed, subsequent to the first image, in which the target tissue is visible.
[0081] Step 306 may include tracking the target tissue in US images subsequent to the first image in the ultrasound image feed. For example, the target tissue may be tracked between successive US images in the US image feed, and the target tissue may be highlighted in each US image subsequent to the first US image in which the target tissue is detected. Thus, the target tissue is highlighted in a sequence of images in which the target tissue can be consistently detected and tracked between one image and the next image in the sequence.
[0082] The skilled person will be familiar with methods of tracking an object from one image to another in a sequence of images. For example, an object may be tracked from one image to another in a sequence if the centers of the detection results in the corresponding images are within a predefined range (in other words, if the centers of the detection results in the corresponding images appear at approximately the same location in the successive images).
[0083] In some embodiments, the step of causing the display to highlight the target tissue in each consecutive US image in the US image feed, following the first image, in which the target tissue is visible, is performed even if the target tissue is visible in one or more consecutive US images with a visibility below a first visibility threshold.
[0084] For example, the target tissue may be highlighted in each successive image in which the target tissue is detected above a second visibility threshold, the second visibility threshold being lower than the first visibility threshold.
[0085] In an embodiment where the target tissue is detected using ML, a step may be performed whereby the display highlights the target tissue in each successive US image in the US image feed, subsequent to the first image, in which the target tissue is visible, even if the ML model detects the target tissue in one or more successive US images with a confidence level below a threshold confidence level.
[0086] In this way, detection results of target tissue can be tracked and highlighted between multiple US images in a US image feed, even if the confidence or visibility between the multiple images varies. The visibility may vary, for example, due to viewing angle or viewing depth, but using the method 300 herein, as long as the target tissue is detected in the first US image with a visibility above a threshold, the target tissue is highlighted in a continuous manner to help the user track the target tissue across different US images in the US image feed.
[0087] In some embodiments, the display may display the US image feed in both a main window of the display and a secondary window of the display. Thus, the step of highlighting the target tissue in the first image on the display may be performed in the US image feed displayed in the secondary window. Figure 5 This situation is illustrated in FIG, where the main window corresponds to the main display area and the auxiliary window corresponds to the live detection area.
[0088] Turning now to other embodiments, in some embodiments, the method 300 may further include grouping the first image and subsequent US images subsequent to the first image in which the target tissue is visible together as a first set of images.
[0089] Furthermore, the target tissue may be tracked in one or more consecutive US images immediately preceding the first US image in which the target tissue is visible (e.g. at a first visibility threshold), and these US images may also be added to the first set of images. In this way, although a detection result above the first visibility threshold triggers highlighting of the target tissue, a sequence of US images in which the target tissue is continuously visible in the US image feed may also be grouped to form the first set of images. The first set of images thus forms a clip or extraction from the complete US image feed.
[0090] The first set of images may be stored for later retrieval and viewing by a user.
[0091] exist Figure 4 This situation is illustrated in the figure. Figure 4 An embodiment of the method 300 is shown. In this embodiment, there is a US image feed including images N to N+6. US images N to N+6 show target tissue 402. In step 302 of the method 300 in this embodiment, the target tissue 402 is detected using an ML model. The target tissue 402 can be detected in a first US image with a confidence level above a first threshold (indicated by a dashed line 404), Figure 4The target tissue is detected in sequential subsequent images N+3-N+6 (image 408) and is highlighted even though the target tissue is detected with a confidence level lower than the first threshold confidence level 404 in images N+3, N+4, and N+6.
[0092] Target tissue is also detected in images N and N+1, and these images are added to image 408 to form a first set of images 410 (eg, a montage) that can be played back to a user upon request.
[0093] Therefore, in this way, the detection results on the consecutive images are linked into a group (the linking method is not limited, for example, the centers of the marks of two consecutive detection results are within a predefined range), and the maximum prediction confidence of the detection results in the group will be used as the prediction confidence of the group. With this setting, during US scanning, once the detection result exceeds the first threshold confidence level 404, the detection results on the subsequent consecutive images are all pushed to the user and these detection results are visible in the live detection area. In addition, as described below, if the target tissue detection result is saved in the detection result buffer, all images containing the detection results in the first group of images 410 can be saved together.
[0094] In some embodiments, the US image feed is cached, for example, US image feeds at predetermined intervals can be kept in a cache (for example, the last x minutes can be cached regardless of the image content), and upon a request received from a user, a portion of the cached US image feed can be replayed to the user, wherein the target tissue in the first set of images is highlighted in the replay.
[0095] In some embodiments, when a user begins to replay a stored set of images (e.g., a stored clip), image acquisition may be paused and the cache frozen and stored for user replay, and the time period corresponding to the stored set of images / clip may be marked on a time bar (e.g., in the form of a slide bar). In other words, upon receiving a request from the user, the display may be caused to replay a portion of the cached US image feed, and the time period in the portion of the cached US image feed may be highlighted on the time bar corresponding to (the temporal position of) the first set of images. This may help the user quickly jump to these images for further review.
[0096] When resuming the US scan, the cache can be cleared because the user has reviewed all the test results that are worth checking.
[0097] In this way, a user can easily locate detection results in the cached US image feed, and replay portions of the US image feed to review both the feed and the output of the automated detection process in order to re-evaluate the detection results.
[0098] It will be appreciated that the above-described method 300 may be performed for more than one target tissue detection result in each US image, and may be repeated for different target tissue detection results in different portions of US images in a US image feed.
[0099] Now turn to Figure 5 , Figure 5 The user interface 500 according to some embodiments of the present invention is illustrated. In this embodiment, there are three zones: a main display area 502, a live detection area (or mirror window) 504, and a detection result buffer 506. In this embodiment, the collected US image feed is displayed on the main display area 502, and no detection result mark is attached thereto. Once the target tissue is detected, a notification message is displayed at the bottom of the main display area 502 to prompt the user that there is a detection result on the image. At the same time, a mark pointing to the detection result is displayed on the live detection area 504, and all successive images containing the target tissue are automatically saved in the detection result buffer. If the user has noticed the target tissue, the user can only ignore the message and continue scanning. Otherwise, the user can check the latest detection result on the live detection result mark or review the detection result buffer 506 to quickly confirm whether the detection result is worthy of further review.
[0100] Figure 6 Another user interface that can be used in conjunction with the above-described user interface 500 is shown. Figure 5 , the user can press the "freeze" button on the ultrasound machine to pause image acquisition. At this point, hundreds of acquired images (e.g., images acquired before pressing "freeze") are saved in the frame buffer and displayed on the main display area 502. The user interface can have a slider (e.g., a time bar) 602 at the bottom, in which a group (or groups) of images with test results are highlighted (e.g., in different colors) to help the user quickly jump to these frames for further review. As described above, the user can press the "freeze" button again to exit the review mode and continue scanning, after which the test result cache can be cleared because the user has reviewed all the test results worth checking.
[0101] Turning to other embodiments, in another embodiment, a computer program product is provided comprising a computer-readable medium having computer-readable code embodied therein, the computer-readable code being configured to cause the computer or processor to perform the method(s) described herein when executed by a suitable computer or processor.
[0102] Thus, it will be appreciated that the present disclosure also applies to computer programs, in particular computer programs on or in a carrier, suitable for putting the embodiments into practice. The program may be in the form of source code, object code, a code intermediate source and object code (e.g. in partially compiled form, or in any other form suitable for implementation of the method according to the embodiments described herein).
[0103] It will also be appreciated that such a program can have many different architectural designs. For example, the program code implementing the function of the method or system can be subdivided into one or more subroutines. Many different ways of allocating functions between these subroutines will be apparent to those skilled in the art. These subroutines can be stored together in an executable file to form a self-contained program. Such an executable file can include computer executable instructions, such as processor instructions and / or interpreter instructions (e.g., Java interpreter instructions). Alternatively, one or more or all of the subroutines in the subroutine can be stored in at least one external library file and, for example, statically or dynamically linked to the main program at runtime. The main program contains at least one call to at least one of the subroutines in the subroutine. The subroutines can also include function calls to each other.
[0104] The carrier of a computer program may be any entity or device capable of carrying the program. For example, the carrier may include a data storage device, such as a ROM (e.g., a CD ROM or a semiconductor ROM) or a magnetic recording medium (e.g., a hard disk). In addition, the carrier may be a transmissible carrier (e.g., an electrical or optical signal), which may be transmitted via an electrical or optical cable or by radio or other means. When the program is embodied in such a signal, the carrier may be constituted by such a cable or other device or means. Alternatively, the carrier may be an integrated circuit in which the program is embedded, the integrated circuit being suitable for executing or for executing the relevant method.
[0105] Those skilled in the art can understand and implement variations of the disclosed embodiments when practicing the principles and techniques described herein by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "one" or "an" do not exclude multiple. A single processor or other unit can implement the functions of several items recorded in the claims. Although certain measures are recorded in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used advantageously. The computer program may be stored or distributed on a suitable medium, for example, an optical storage medium or solid-state medium supplied together with other hardware or as part of other hardware, but may also be distributed in other forms (for example, distributed via the Internet or other wired or wireless telecommunications systems). Any figure mark in the claims should not be interpreted as limiting the scope.
Claims
1. A computer implemented method of analyzing an ultrasound image feed of a region of interest as part of an ultrasound (US) examination, said US image feed not being a time series of contrast enhanced ultrasound images, said method comprising: detecting (302) a target tissue in a first image of the US image feed; If the target tissue has a visibility above a first visibility threshold, sending (304) a message to a display to cause the display to highlight the target tissue in the first image on the display; and The display is caused (306) to highlight the target tissue in each successive US image in the US image feed, subsequent to the first image, in which the target tissue is visible.
2. The method according to claim 1, wherein: Even if the target tissue is visible in one or more consecutive US images with a visibility below the first visibility threshold, a step of causing (306) the display to highlight the target tissue in each consecutive US image in the US image feed that follows the first image in which the target tissue is visible is performed.
3. The method according to claim 1 or 2, wherein: The step of detecting (302) target tissue in a first image of the US image feed is performed using a machine learning model trained to detect target tissue in US images.
4. The method according to claim 3, wherein: The first visibility threshold is a threshold confidence level, and the step of sending (304) a message to a display to cause the display to highlight the target tissue in the first image on the display is performed in response to the machine learning model detecting the target tissue with a confidence level above the threshold confidence level.
5. A method according to any one of the preceding claims, wherein: The method further comprises grouping the first image and the consecutive US images subsequent to the first image in which the target tissue is visible together as a first set of images.
6. The method according to claim 5, wherein: The first set of US images also includes consecutive US images immediately preceding the first image in which the target tissue is visible below the first visibility threshold.
7. The method according to claim 5 or 6, further comprising: caching the US image feed; and Upon receiving a request from a user, the display is caused to replay a portion of the cached US image feed and highlight a time period in the portion of the cached US image feed on a time bar corresponding to the first set of images.
8. A method according to any one of the preceding claims, wherein: The method is performed in real time on a live ultrasound image feed.
9. The method according to claim 8, wherein: The step of sending (304) a message to a display causes the display to display an alert that a live detection has occurred.
10. The method of any preceding claim, further comprising causing the display to: displaying the US image feed in a main window of the display; displaying the US image feed in a secondary window of the display; and wherein, The step of highlighting the target tissue in the first image on the display is performed in the US image feed displayed in the secondary window.
11. A method according to any one of the preceding claims, wherein: The US image feed comprises a sequence of ultrasound images acquired while the relative spatial position between the field of view of the ultrasound image and the region of interest is varied.
12. A method according to any one of the preceding claims, wherein: The region of interest includes breast tissue and the US image feed includes images of the breast tissue obtained as part of an examination for breast lesions.
13. An apparatus for analysing an ultrasound image feed of a region of interest as part of an ultrasound (US) examination, the US image feed not being a time series of contrast-enhanced ultrasound images, the apparatus being configured to perform a method according to any one of the preceding claims.
14. An ultrasound (US) imaging system comprising the apparatus according to claim 13 and a display for displaying the ultrasound image feed; and wherein, The message and further messages cause the display to highlight the target tissue in the first image and the target tissue in the subsequent US images on the display.
15. A computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that when executed by a suitable computer or processor, the computer or processor is caused to perform the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Phased array acoustic systems with intra-group processors
US5997479A
Beamforming methods and apparatus for three-dimensional ultrasound imaging using two-dimensional transducer array
US6013032A
Ultrasonic diagnostic imaging with blended tissue harmonic signals
US6283919B1
Method for creating multiplanar ultrasonic images of a three dimensional object
US6443896B1
Ultrasonic harmonic imaging with adaptive image formation
US6458083B1
