predicting the likelihood that an individual has one or more pathologies

Abstract

A mechanism for identifying whether one or more lesions are present in a plurality of ultrasound images. The ultrasound images are processed to identify regions of interest in each image, each region of interest representing a portion of the ultrasound image suspected of depicting a lesion. Combinations of similar regions of interest are brought together and then processed using a machine learning method to predict whether they contain a lesion of interest. Thus, a two-step process for identifying lesions of interest is performed.

Description

Technical Field

[0001] This invention relates to the field of ultrasound imaging, and more particularly to the identification of lesions represented in ultrasound images. Background Technology

[0002] Ultrasound is increasingly used for the detection and diagnosis of various cancers, such as breast cancer, thyroid cancer, and liver cancer. In particular, ultrasound is often used for early detection of cancer by identifying lesions in ultrasound images. Unfortunately, the detection rate of early-stage cancer is relatively low due to a number of factors, such as a lack of experienced sonographers and time pressure in performing ultrasound imaging screenings.

[0003] Clinical decision support (CDS) systems, employing computer-based ultrasound lesion detection technology, are being used to assist clinicians in ultrasound image screening. Such systems can support clinicians by predicting the presence or absence of one or more lesions, thereby helping to reduce the likelihood of missed diagnoses and improving diagnostic confidence as a double read / confirmation.

[0004] Generally, there are two types of ultrasound-based lesion detection methods (automatic or computer-implemented): traditional lesion detection methods based on image processing steps and deep learning methods. For most people, traditional methods are generally considered less robust and / or flexible because they rely on rule-based approaches and specific assumptions. Deep learning methods exhibit higher accuracy in object / lesion detection because they rely less on such strong assumptions, but they have the drawbacks of high computational complexity and long processing time.

[0005] In their paper "3D tumor detection in automated breast ultrasound using deep convolutional neutral network," published in Medical Physics (vol. 47, no. 11), LI YANFENG et al. proposed a method for tumor detection using automated breast ultrasound (ABUS). ABUS is described as an automated ultrasound device for breast screening. ABUS uses 3D ultrasound technology to acquire operator-independent breast volume images and generates and displays coronal view slices based on the acquired 3D volume.

[0006] Therefore, there has been a long-standing desire to improve mechanisms for predicting the likelihood of an individual having one or more lesions. Summary of the Invention

[0007] This invention is defined by the claims.

[0008] According to an example of one aspect of the invention, a processing system is provided for predicting the likelihood that an individual has one or more lesions. The processing system is configured to: acquire multiple ultrasound images of the individual; identify regions of interest (ROIs) in the multiple ultrasound images, each ROI being a portion of an ultrasound image representing a region with a potential lesion in the individual; group the ROIs from different ultrasound images from the multiple ultrasound images together based on the similarity of the ROIs; and process each group of ROIs using a machine learning method to generate a prediction indicator indicating the likelihood that the group of ROIs in the individual contains a lesion. The multiple ultrasound images include a time series of ultrasound images. In some embodiments, the time series of ultrasound images is a video of ultrasound images, such as a movie playback. Identifying ROIs representing regions with potential lesions includes identifying the presence or absence of one or more ROIs. For some ultrasound images, ROIs may not be identifiable in every ultrasound image; for some ultrasound images, one or more ROIs may be identifiable in every ultrasound image.

[0009] This disclosure proposes a method for identifying the presence or absence of one or more lesions in an individual, and particularly for generating one or more predictive indicators indicating the likelihood of the presence of a lesion. Ultrasound images of a patient are processed to identify regions of interest (i.e., portions of each ultrasound image) containing potential lesions—that is, candidate regions predicted to depict lesions. Thus, a region of interest is a portion of an ultrasound image and therefore an image in itself (albeit smaller in size than the ultrasound image). If a region of an individual is predicted to contain a potential lesion—that is, likely to contain a lesion—then the region of interest contains the potential lesion. Mechanisms for identifying regions of interest containing potential lesions are well known to those skilled in the art, and may employ, for example, machine learning methods, edge detection algorithms, image segmentation techniques, etc.

[0010] The regions of interest (ROIs) are then grouped or clustered based on their similarity to create groups of ROIs. In this way, portions of different ultrasound images that correspond to each other (i.e., contain the same underlying lesion) are grouped together. Therefore, if the ROIs are predicted to contain the same underlying lesion in individuals, they can be considered similar to each other (and grouped accordingly). Furthermore, the grouping or clustering is based on the similarity of the ROIs, without requiring spatial relationships between them. This allows grouping or clustering to be performed without spatially aligning multiple ultrasound images. This is particularly advantageous when multiple ultrasound images are spatially misaligned. For example, in the case of manually acquiring multiple two-dimensional ultrasound images with a handheld ultrasound probe, the relative spatial relationships between these two-dimensional ultrasound images are different from those acquired by ABUS (Automated Breast Ultrasound) and are unknown. Traditionally, such multiple ultrasound images can first be combined into a three-dimensional (3D) volume using various spatial alignment techniques (e.g., spatial registration, 3D construction), and then further signal processing (e.g., lesion detection) can be performed based on the spatially aligned images or slices. However, in some scenarios, spatial alignment or 3D can be challenging. Instead, this paper proposes grouping multiple regions of interest (ROIs) based on the similarity between them, without relying on any pre-spatial registration or alignment across multiple images, or in other words, without relying on any knowledge of the relative spatial relationships between the multiple images or the ROIs identified in the multiple images.

[0011] These groups are then processed using machine learning methods to generate predictive indicators that indicate the likelihood that the group contains the individual's lesion. Thus, the machine learning method effectively predicts the presence of lesions within a group of regions of interest. In this way, the machine learning method (e.g., a classifier) ​​generates predictive indicators by processing a set of regions of interest containing the same potential lesions.

[0012] In the context of this invention, a prediction indicator is any data that changes in response to changes in the predicted probability (determined by a machine learning method) that a lesion group contains a lesion. Prediction indicators can include binary, categorical, or numerical data. Binary data can indicate a prediction about whether the group contains a lesion (e.g., "0" indicates a prediction of no lesion, "1" indicates a prediction of presence, and vice versa). Categorical data can indicate the probability category of the group containing a lesion (e.g., "likely," "very likely," "unlikely," etc.). Numerical data can indicate the numerical probability that a set of regions of interest contains a lesion, for example, in the range of 0 to 1 or 0 to 100.

[0013] The proposed method reduces the false positive rate of identifying individual lesions (e.g., from still images alone) by using multiple ultrasound images (e.g., taken over a period of time), thereby leveraging additional contextual information to identify one or more lesions in an individual. By first identifying the region of interest, the amount of processing performed to identify the lesion is reduced compared to performing a machine learning process on all multiple ultrasound images without further processing. Therefore, the proposed method provides a mechanism for performing high-quality and high-accuracy lesion identification while reducing computational cost.

[0014] Preferably, each ultrasound image is a two-dimensional ultrasound image. This invention is particularly advantageous when used for processing two-dimensional ultrasound images because ultrasound systems for generating 2D ultrasound images have a wide range of applications and are increasingly used in resource-constrained environments, such as low-power environments (e.g., battery-powered or areas with unstable power). Therefore, reducing the computational burden (to minimize power usage requirements) while still obtaining an accurate indication of the likelihood of pathology in such systems would be especially beneficial.

[0015] In some embodiments, the processing system is configured to process each group of regions of interest by performing the following process, the process including: for each group of regions of interest: generating a sequence of regions of interest using regions of interest in the group of regions of interest; processing the sequence of regions using machine learning methods to predict whether the sequence of regions contains lesions of an individual.

[0016] The sequence of regions of interest can be efficiently formed into a single data structure containing data from all regions of interest in the group.

[0017] The processing system can be configured to generate a sequence of regions by performing steps including stacking regions of interest. If the regions of interest are two-dimensional images, such as those extracted from two-dimensional ultrasound images, the process effectively forms pseudo-3D volumes.

[0018] The processing system can be configured to identify regions of interest (ROIs) in ultrasound images by performing a process that includes processing each ultrasound image using a second machine learning method to identify ROIs. Therefore, at least two machine learning methods can be used to predict the presence of lesions. This approach increases the likelihood of identifying potential lesions and facilitates fully automated mechanisms for identifying potential lesions, as well as the use of existing and well-established mechanisms to improve reliability in identifying potential lesions in ultrasound images.

[0019] In some examples, a third machine learning method is used to group regions of interest, such as grouping regions of interest based on the characteristics and / or other metadata (e.g., location, size, shape, etc.) of each region of interest. Of course, machine learning methods can also be used to group regions of interest even without using them to identify them.

[0020] In some preferred examples, the multiple ultrasound images comprise a sequence of ultrasound images. The sequence of ultrasound images is preferably a time series, such that later ultrasound images in the sequence were acquired by the imaging system later than earlier ultrasound images. This effectively results in time-related information being incorporated into the group of regions of interest. This disclosure recognizes that the use of time-based information increases the accuracy of identifying changes in the presence or absence of one or more lesions.

[0021] Preferably, multiple ultrasound images are acquired using the same ultrasound imaging system, such as using the same ultrasound imaging probe. More preferably, multiple images are acquired while the ultrasound imaging probe (used to acquire ultrasound images) is stationary. This increases the ease and accuracy of identifying groups of linked regions of interest.

[0022] In some embodiments, the order of the ultrasound images depends on the time at which each ultrasound image was captured. For example, the sequence of ultrasound images may include sequential frames of an ultrasound video. The context provided by the temporally ordered sequence of ultrasound images (e.g., ultrasound video) allows machine learning methods to consider time-based information when predicting the presence or absence of one or more lesions in the group of regions of interest. This disclosure determines that this information is particularly beneficial for reducing the false positive detection rate of one or more lesions in ultrasound images.

[0023] In some examples, each set of regions of interest, each region of interest is derived from ultrasound images sequentially adjacent to ultrasound images of another region of interest within the same set of regions of interest.

[0024] Preferably, each region of interest is no larger than 0.4 times the size of the ultrasound image, for example, no larger than 0.25 times the size of the ultrasound image. The smaller the size of the region of interest, the greater the reduction in computational complexity of processing a set of regions of interest (rather than the entire ultrasound image).

[0025] In some examples, the processing system is configured to display visually perceptible output at a display device in response to each prediction indicator. Therefore, information in response to the prediction indicators can be output to a user (e.g., a clinician).

[0026] The processing system can be configured to: if at least one prediction indicator indicates that at least one set of regions of interest may contain a lesion; identify the location of the identified lesion relative to the displayed ultrasound image of the patient; and, in response to the identified location of the lesion, superimpose a visually perceptible output onto the displayed ultrasound image.

[0027] The processing system can be configured to group regions of interest by performing a process that includes: determining a similarity metric between different regions of interest from different ultrasound images, the similarity metric representing the size of the overlap between a relative area occupied by one region of interest in one ultrasound image and a relative area occupied by another region of interest in another ultrasound image; and grouping the two different regions of interest into the same group in response to the similarity metric between the two different regions of interest exceeding a predetermined threshold.

[0028] Therefore, similarity measurements can be based on the size of the overlap, such as the intersection-over-union (IoU) metric between regions of interest from different ultrasound images. This facilitates a simple mechanism for identifying whether the same underlying lesion is being identified, for example, since the same underlying lesion would likely be located at the same relative position in each ultrasound image. This method is particularly advantageous if multiple ultrasound images comprise frames from an ultrasound video, as this embodiment increases the likelihood that underlying lesions will be located at the same position across different frames of the ultrasound video.

[0029] The processing system can be configured to group regions of interest by performing a process that includes: identifying metadata of the regions of interest, the metadata providing information about the size, location, confidence level, and / or appearance of the regions of interest; and grouping the regions of interest based on the metadata. Therefore, additional information can be used to determine the similarity between different regions of interest. In some embodiments, machine learning methods are used to group regions of interest, for example, based on the metadata of the regions of interest.

[0030] A computer-based method for predicting the likelihood of an individual having one or more lesions is also proposed.

[0031] The computer-implemented method includes: acquiring multiple ultrasound images of an individual; identifying regions of interest (ROIs) in the multiple ultrasound images, each ROI being a portion of an ultrasound image representing a potential lesion in the individual; grouping ROIs from different ultrasound images of the multiple ultrasound images together based on the similarity of the ROIs; and processing each group of ROIs using a machine learning method to generate a prediction indicator indicating the likelihood that the group of ROIs in the individual contains a lesion.

[0032] A processing system is also proposed, which includes a memory and a processor coupled to the memory and configured to perform such a computer-implemented method.

[0033] A computer program product comprising computer program code units is also proposed, which, when run on a computing device having a processing system, cause the processing system to perform all the steps of any computer implementation method described herein.

[0034] Those skilled in the art will be able to readily employ any processing system described herein to perform any of the methods described herein, and vice versa.

[0035] These and other aspects of the invention will become apparent and will be explained with reference to one or more embodiments described below. Attached Figure Description

[0036] To better understand the invention and to more clearly illustrate how it can be practiced, reference will now be made to the accompanying drawings by way of example only, wherein,

[0037] Figure 1 The illustration shows an ultrasound system used in one embodiment;

[0038] Figure 2 The diagram illustrates the workflow for understanding the embodiments in context;

[0039] Figure 3 The method according to an embodiment is illustrated;

[0040] Figure 4 The method according to an embodiment is illustrated;

[0041] Figure 5 The diagram illustrates a method for grouping regions of interest.

[0042] Figure 6 The illustration shows a processing system according to an embodiment;

[0043] Figure 7 The illustrated embodiment shows the effect; and

[0044] Figure 8 The diagram illustrates the processing system. Detailed Implementation

[0045] The invention will be described with reference to the accompanying drawings.

[0046] It should be understood that while the detailed description and specific examples indicate exemplary embodiments of the devices, systems, and methods, they are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0047] This invention provides a mechanism for determining the probability / likelihood of the presence / absence of one or more lesions in an individual by processing multiple ultrasound images of that individual. The ultrasound images are processed to identify regions of interest (ROIs) in each image, each ROI representing a portion of the ultrasound image suspected of depicting a lesion. Similar ROIs are grouped together, and the groups of similar regions are then processed using machine learning methods to generate predictions about whether they are likely to contain the lesion of interest. Thus, a two-step process for identifying the likelihood of a lesion is performed.

[0048] The embodiments of the present invention are based on the understanding that if only a single ultrasound image is used to identify potential lesions, important information is lost or not present, but processing multiple ultrasound images simultaneously significantly increases the processing burden of the image analysis process. Instead, a novel method is proposed for assessing the probability of the presence and / or absence of potential lesions, wherein groups of regions of interest from different ultrasound images are formed and processed together to predict the probability of the presence of a lesion.

[0049] The embodiments can be employed in any suitable ultrasound analysis system, for example, in ultrasound analysis used for cancer screening and superficial organ diagnosis. The proposed embodiments are likely to be employed in any suitable clinical setting, such as during health checks for cancer screening, or in a hospital ultrasound department for cancer screening and diagnosis. This invention is applicable to all ultrasound imaging systems with advanced lesion recognition software for online or offline use, i.e., processing data from memory or data generated directly from the ultrasound system.

[0050] Now, the first reference Figure 1 The general operation of an exemplary ultrasound system is described. This invention utilizes ultrasound images generated by such an ultrasound system; however, other methods and systems for generating ultrasound images will be readily apparent to those skilled in the art.

[0051] The system includes an array transducer probe 4 having a transducer array 6 for transmitting ultrasound and receiving echo information. The transducer array 6 may include: a CMUT transducer; a piezoelectric transducer made of materials such as PZT or PVDF; or any other suitable transducer technology. In this example, the transducer array 6 is a two-dimensional array of transducers 8 capable of scanning a 2D plane or a three-dimensional volume of a region of interest. In another example, the transducer array may be a 1D array.

[0052] The transducer array 6 is coupled to a microwave beamformer 12, which controls the signal reception of the transducer elements. As described in U.S. Patents US 5,997,479 (Savord et al.), US 6,013,032 (Savord), and US 6,623,432 (Powers et al.), the microwave beamformer is capable of performing at least partial beamforming on the signals received by the subarrays of the transducers (generally referred to as a “group” or “patch”).

[0053] It should be noted that the microwave beamformer is entirely optional. Furthermore, the system includes a transmit / receive (T / R) switch 16, to which the microwave beamformer 12 can be coupled and used to switch the array between transmit and receive modes, and to protect the main beamformer 20 from high-energy transmitted signals when the microwave beamformer is not used and the transducer array is directly operated by the main system beamformer. The transmission of the ultrasonic beam from the transducer array 6 is indicated via the T / R switch 16 to a transducer controller 18 of the microwave beamformer and the main transmitting beamformer (not shown), which can receive input from user-to-user interface or control panel 38. The controller 18 may include transmit circuitry arranged to drive the transducer elements of array 6 during transmit mode (directly or via the microwave beamformer).

[0054] In a typical line-by-line imaging sequence, the beamforming system within the probe can operate as follows. During transmission, a beamformer (which may be a microwave beamformer or a main system beamformer, depending on the implementation) activates the transducer array or sub-apertures of the transducer array. The sub-apertures can be one-dimensional lines of transducers within a larger array or two-dimensional sheets of transducers. In transmission mode, the focusing and steering of the ultrasonic beam generated by the array or its sub-apertures are controlled as described below.

[0055] Upon receiving the backscattered echo signal from the object, the received signal undergoes receive beamforming (as described below) to align the received signal, and, in the case of a sub-aperture, the sub-aperture is shifted, for example, by a transducer element. The shifted sub-aperture is then activated, and this process is repeated until all transducer elements of the transducer array are activated.

[0056] For each line (or sub-aperture), the total received signal associated with the line used to form the final ultrasound image will be the sum of the voltage signals measured by the transducer elements of a given sub-aperture during the reception period. Following the beamforming process below, the resulting line signals are typically referred to as radio frequency (RF) data. Each line signal (RF dataset) generated from the individual sub-apertures is then further processed to generate the line of the final ultrasound image. The variation in the amplitude of the line signal over time contributes to the variation in brightness of the ultrasound image with depth, where high-amplitude peaks will correspond to bright pixels (or sets of pixels) in the final image. Peaks appearing near the beginning of the line signal will represent echoes from shallow structures, while peaks appearing later in the line signal will represent echoes from structures of increasing depth within the object.

[0057] One of the functions controlled by the transducer controller 18 is the direction in which the beam is steered and focused. The beam can be steered vertically forward from the transducer array (perpendicular to the transducer array) or at different angles for a wider field of view. The steer and focus of the transmitted beam can be controlled based on the actuation time of the transducer elements.

[0058] In general ultrasound data acquisition, two methods can be distinguished: plane wave imaging and beam steering imaging. The difference between the two methods lies in the presence of beamforming in the transmit mode (beam steering imaging) and / or receive mode (plane wave imaging and beam steering imaging).

[0059] First, let's look at the focusing function. By simultaneously activating all transducer elements, the transducer array generates a plane wave that diverges as it passes through an object. In this case, the ultrasound beam remains unfocused. By introducing a location-dependent time delay into the transducer activation, the wavefront of the beam can be converged to a desired point, called the focal zone. The focal zone is defined as a point where the lateral beamwidth is less than half the width of the emitted beam. In this way, the lateral resolution of the final ultrasound image is improved.

[0060] For example, if a time delay causes the transducer elements to activate sequentially, starting from the outermost element and ending at one or more central elements of the transducer array, a focal region will be formed at a given distance from the probe, aligned with the central elements. The distance between the focal region and the probe will vary depending on the time delay between each subsequent round of transducer element activation. After the beam passes through the focal region, it will begin to diverge, forming a far-field imaging region. It should be noted that for the focal region located close to the transducer array, the ultrasound beam will diverge rapidly in the far field, resulting in beamwidth artifacts in the final image. Typically, due to large overlap in the ultrasound beam, little detail is displayed in the near field between the transducer array and the focal region. Therefore, changing the position of the focal region can lead to significant changes in the final image quality.

[0061] It should be noted that in emission mode, only one focus can be defined unless the ultrasound image is divided into multiple focal regions (each of which may have a different emission focus).

[0062] Furthermore, upon receiving an echo signal from within the object, the reverse process described above can be performed to achieve receiver focusing. In other words, the incoming signal can be received by the transducer elements and undergoes an electronic time delay before being transmitted to the system for signal processing. The simplest example is called delay-summation beamforming. The receiver focusing of the transducer array can be dynamically adjusted according to time.

[0063] Now consider the function of beam steering. By correctly applying a time delay to the transducer elements, a desired angle can be imparted to the ultrasonic beam as it leaves the transducer array. For example, by activating the transducers on the first side of the transducer array, and then activating the remaining transducers in a sequence ending on the opposite side of the array, the wavefront of the beam will tilt towards the second side. The size of the steering angle relative to the normal of the transducer array depends on the size of the time delay between the subsequent activation of the transducer elements.

[0064] Furthermore, the steering beam can be focused, where the total time delay applied to each transducer element is the sum of the focusing and steering time delays. In this case, the transducer array is called a phased array.

[0065] When the CMUT transducer requires a DC bias to be activated, the transducer controller 18 can be coupled to control the DC bias controller 45 of the transducer array. The DC bias controller 45 sets the bias applied to one or more of the CMUT transducer elements.

[0066] For each transducer element in the transducer array, an analog ultrasonic signal (typically referred to as channel data) enters the system through a receiving channel. In the receiving channel, a partially beamformed signal is generated from the channel data by a microwave beamformer 12 and then transmitted to a main receiving beamformer 20. In the main receiving beamformer 20, the partially beamformed signals from individual transducer elements are combined into a fully beamformed signal, referred to as radio frequency (RF) data. Beamforming at each stage can be performed as described above, or additional functionality can be included. For example, the main beamformer 20 may have 128 channels, each receiving tens or hundreds of partially beamformed signals from the transducer elements. In this way, signals received by thousands of transducers in the transducer array can be effectively contributed to a single beamformed signal.

[0067] The beamformed received signal is coupled to signal processor 22. Signal processor 22 can process the received echo signal in various ways, such as: bandpass filtering; decimation; I and Q component separation; and harmonic signal separation to separate linear and nonlinear signals in order to identify nonlinear (higher harmonics of the fundamental frequency) echo signals returning from tissue and microbubbles. The processor can also perform signal enhancement, such as ripple reduction, signal recombination, and noise cancellation. The bandpass filter in the signal processor can be a tracking filter, wherein its passband slides from a higher frequency band to a lower frequency band as the echo signal is received from increasing depth, thereby rejecting noise from higher frequencies at greater depths, which typically lacks anatomical information.

[0068] Beamformers for transmitting and receiving are implemented in different hardware and can have different functions. Of course, the receiver beamformer is designed to take into account the characteristics of the transmitting beamformer. For simplicity, in Figure 1 Only receiver beamformers 12 and 20 are shown in the diagram. Throughout the system, there will also be a transmitter chain with a transmit microwave beamformer and a main transmit beamformer.

[0069] The function of microwave beamformer 12 is to provide an initial combination of signals in order to reduce the number of analog signal paths. This is typically performed in the analog domain.

[0070] The final beamforming is performed in the main beamformer 20, and typically after digitization.

[0071] The transmit and receive channels use the same transducer array 6 with a fixed frequency band. However, the bandwidth occupied by the transmit pulse can vary depending on the transmit beamforming used. The receive channel can capture the entire transducer bandwidth (this is the classic approach), or it can extract only the bandwidth containing the desired information (e.g., harmonics of the main harmonic) by using bandpass processing.

[0072] The RF signal can then be coupled to a B-mode (i.e., luminance mode or 2D imaging mode) processor 26 and a Doppler processor 28. The B-mode processor 26 performs amplitude detection on the received ultrasound signal to image structures in the body, such as organs, tissues, and blood vessels. In the case of line-by-line imaging, each line (beam) is represented by an associated RF signal whose amplitude is used to generate a luminance value to be assigned to a pixel in the B-mode image. The exact location of a pixel within the image is determined by its position along the associated amplitude measurement of the RF signal and the number of lines (beams) of the RF signal. B-mode images of this configuration can be formed in harmonic or fundamental image modes or a combination of both, as described in U.S. Patent 6283919 (Roundhill et al.) and U.S. Patent 6458083 (Jago et al.). The Doppler processor 28 can process temporally discrete signals originating from tissue motion and blood flow for detecting moving substances, such as the flow of blood cells in the image field. Doppler processor 28 typically includes a wall filter having parameters that are set to allow or reject echoes returning from a selected type of material in the body.

[0073] The structural and motion signals generated by the B-mode and Doppler processors are coupled to a scan converter 32 and a multiplane reformer 44. The scan converter 32 arranges the echo signals in the desired image format according to the spatial relationships in which the echo signals are received. In other words, the scan converter converts the RF data from a cylindrical coordinate system to a Cartesian coordinate system suitable for displaying an ultrasound image on an image display 40. In the case of B-mode imaging, the brightness of a pixel at a given coordinate is proportional to the amplitude of the RF signal received from that location. For example, the scan converter can arrange the echo signals in a two-dimensional (2D) sector format or a cone-shaped three-dimensional (3D) image. The scan converter can superimpose colors corresponding to the motion of individual points in the image field onto the B-mode structural image, where the Doppler-estimated velocity produces the given color. The combined B-mode structural image and color Doppler image describe the motion of tissue and blood flow within the structural image field. The multiplane reformer converts echoes received from points in a common plane within a volumetric region of the body into an ultrasound image of that plane, as described in U.S. Patent US 6443896 (Detmer). Volume plotter 42 converts the echo signal of the 3D dataset into a 3D image of the projection as seen from a given reference point, as described in U.S. Patent US 6530885 (Entrekin et al.).

[0074] 2D or 3D images are coupled from the scan converter 82, multiplane reformer 44, and volume plotter 42 to the image processor 30 for further enhancement, caching, and temporary storage for display on the image display 40. The image processor can be adapted to remove certain imaging artifacts from the final ultrasound image, such as: acoustic shadowing, e.g., caused by strong attenuators or refraction; post-enhancement, e.g., caused by weak attenuators; reverberation artifacts, e.g., adjacent locations of highly reflective tissue interfaces; and so on. Additionally, the image processor can be adapted to perform specific ripple reduction functions to improve the contrast of the final ultrasound image.

[0075] In addition to being used for imaging, blood flow values ​​generated by the Doppler processor 28 and tissue structure information generated by the B-mode processor 26 are coupled to the quantization processor 34. The quantization processor generates measurements of different flow conditions (e.g., blood flow volume ratio) and structural measurements (e.g., organ size and gestational age). The quantization processor 46 can receive output from the user control panel 38, such as points in the anatomical structures of the image to be measured.

[0076] Output data from the quantization processor is coupled to the graphics processor 36 for reproducing measurement graphs and values ​​on the display 40 along with images, and for outputting audio from the display device 40. The graphics processor 36 can also generate graphic overlays for display alongside ultrasound images. These overlays may include standard identification information such as the patient's name, date and time of the image, imaging parameters, etc. For these purposes, the graphics processor receives input from the user interface 38, such as the patient's name. The user interface is also coupled to the transmit controller 18 to control the generation of ultrasound signals from the transducer array 6, and thus control the images generated by the transducer array and the ultrasound system. The transmit control function of the controller 18 is only one of the functions performed. The controller 18 also considers the operating mode (given by the user) and the corresponding required transmitter and bandpass configurations in the receiver analog-to-digital converter. The controller 18 may be a state machine with fixed states.

[0077] The user interface can also be coupled to the multiplane reformer 44 to select and control the planes of multiple multiplane reformulated (MPR) images, which can be used to perform quantization measurements in the image field of the MPR images.

[0078] This disclosure relates to processing for analyzing multiple ultrasound images, such as those generated from previously described ultrasound images. This process can be performed by an ultrasound image processor, such as a graphics processor 36, an image processor 30, and / or a separate / dedicated processing system (not shown).

[0079] Figure 2The flowchart 200 is schematically illustrated for understanding the methods employed in embodiments of the present invention. The workflow represents the process executed by the processing system.

[0080] This workflow involves processing multiple ultrasound images 210, all having the same resolution. These multiple ultrasound images are individual ultrasound images, and specifically, ultrasound images of the same anatomical region of an individual (and preferably acquired from the same or similar viewpoints, e.g., using a substantially stationary imaging probe). Preferably, each ultrasound image is a two-dimensional image, but this is not required (e.g., 3D ultrasound images can be used).

[0081] An ultrasound image is any image that has been obtained using an ultrasound imaging process, such as using an ultrasound imaging system as previously described. In some examples, the “raw” ultrasound image undergoes additional processing (e.g., using one or more filters and / or one or more layers of a neural network) before being used as an ultrasound image for the purpose of the method / workflow. Therefore, an ultrasound image can be a feature space image or an image space ultrasound image.

[0082] In a particularly preferred example, the multiple ultrasound images comprise a sequence of ultrasound images, such as a sequence of ultrasound images captured at different times by the same ultrasound imaging probe. Preferably, the order of the ultrasound image sequence depends on the time at which each ultrasound image was captured. For example, the sequence may comprise sequential frames of ultrasound video.

[0083] In process 220, each of the multiple ultrasound images is processed to identify a region of interest within each ultrasound image. A region of interest is a portion or section of an ultrasound image (i.e., not the entire ultrasound image) that represents an area with / containing a potential lesion in an individual.

[0084] The lesion may include one or more of the following: hyperplasia, tumor, abscess, nodule, swelling, bulge, ulcer, or any other suitable feature caused by injury, disease, or damage within an individual. Mechanisms for identifying regions of interest within ultrasound images are well known to those skilled in the art and can employ, for example, machine learning methods, edge detection algorithms, image segmentation techniques, etc.

[0085] Regions of interest 230 then undergo a grouping or clustering process 240, where similar regions of interest are grouped together. In other words, regions of interest are grouped based on the similarity between them (e.g., into one or more groups of regions of interest). The similarity between regions of interest can be defined based on the similarity of their content. Specifically, regions of interest can be grouped together if they are predicted to identify the same underlying lesions in individuals.

[0086] Then, in process 260, machine learning methods are used to process each group of regions of interest 250 (only one example is shown) to generate a prediction indicator 270, which indicates the probability that the group of regions of interest contains the lesion in the individual.

[0087] The prediction indicator 270 may include binary, categorical, or numerical data indicating the probability that the group contains a lesion. As an example, the prediction indicator may be a binary indicator (e.g., "0" or "1") indicating a prediction of whether a group of regions of interest contains a lesion. As another example, the prediction indicator may be the probability (i.e., a numerical indicator) that a group of regions of interest contains a lesion. Numerical indicators may be in the range of 0 to 1, 0 to 10, 1 to 10, 0 to 100, or 1 to 100 (but other examples may be used). As yet another example, the prediction indicator may be a categorical indicator indicating the probability that a group of regions of interest contains a lesion (e.g., "likely", "unlikely", "very likely", "neither likely nor unlikely", etc.).

[0088] Therefore, the proposed method performs a two-step process for predicting the likelihood of a lesion in an individual using multiple ultrasound images. Each ultrasound image is processed individually to identify a region of interest (ROI), which is a region / part of the ultrasound image that is likely to depict a lesion (i.e., as a candidate region containing a lesion). Similar ROIs are then grouped together and processed using machine learning methods to generate predictive indicators in response to the likelihood of the group depicting a lesion. The processing of the group effectively confirms or rejects the suggestion that each ROI in the group depicts a lesion (the suggestion was proposed by the processing of each individual ultrasound image).

[0089] The proposed method avoids the need for complex, high-intensity processing of all ultrasound images simultaneously by processing each individual ultrasound image separately before processing a subset of the ultrasound images. The inventors have recognized that the proposed method can significantly reduce the false positive rate while maintaining high accuracy / precision in predicting the likelihood of lesions.

[0090] Having described a general overview of the workflow, a more complete example of how to execute a model workflow will be provided below.

[0091] Figure 3 The illustration shows a method 300 according to an embodiment. Method 300 can be executed by a processing system according to the embodiment.

[0092] Method 300 includes step 310 of obtaining multiple ultrasound images of an individual. The multiple ultrasound images can be obtained as previously referenced. Figure 2As described. For ease of explanation, the illustrated ultrasound image is a spatial ultrasound image (i.e., each pixel directly represents a portion of an individual's anatomical structure).

[0093] Specifically, the ultrasound images are preferably two-dimensional ultrasound images, but the use of 3D ultrasound images is also possible. Preferably, the multiple ultrasound images comprise a series of ultrasound images, for example, wherein one ultrasound image is captured sequentially after or later than another ultrasound image at a later time point. In some examples, the multiple ultrasound images comprise (sequential) frames from an ultrasound video.

[0094] The ultrasound image is then processed in step 320, which includes identifying one or more regions of interest (ROIs) within the ultrasound image, each ROI being a portion of the ultrasound image representing a region in an individual with a potential lesion. Thus, the ROI itself is a portion of the ultrasound image, which is a part of a larger ultrasound image.

[0095] The processing for identifying regions of interest (ROIs) in ultrasound images will be readily apparent to technicians, and machine learning methods (such as neural networks or Naive Bayes classifiers) can be used to identify any portion of an ultrasound image that contains or represents an individual's potential lesion. Typically, ROIs are defined by a bounding box or volume (with coordinates) that defines the outer boundary of the region or volume containing the individual's potential lesion. Therefore, each ROI is a specific portion of the ultrasound image.

[0096] Preferably, each region of interest is no larger than 0.4 times the size of the ultrasound image, for example, no larger than 0.25 times the size of the ultrasound image. The smaller the region of interest, the greater the reduction in computational complexity for processing a set of regions of interest (compared to the entire ultrasound image).

[0097] Regions of interest (ROIs) can be processed to remove or delete any duplicate ROIs (e.g., ROIs in the same ultrasound image that identify the same underlying lesion). This can be done, for example, by processing each ultrasound image to identify any ROIs in that image that overlap with each other by more than a predetermined amount (e.g., having an intersection value exceeding a predetermined value) and deleting one or more of the overlapping ROIs. The deleted ROI can be the ROI associated with the lowest confidence / probability that it identifies the underlying lesion. Such confidence / probability values ​​can be generated if machine learning methods are used to identify ROIs, but other techniques also generate such confidence / probability metrics.

[0098] Method 300 then performs step 330, grouping or clustering the identified regions of interest to form groups of regions of interest. This grouping is performed based on the similarity between the identified regions of interest. Each group can be modified / configured to include regions of interest that depict / have the same potential lesions.

[0099] Step 330 can be performed by linking or associating regions of interest (ROIs) from different ultrasound images. Linked ROIs are those regions believed to contain the same underlying lesion. Step 330 can then create or form a group of ROIs by grouping or clustering the linked ROIs into single groups, for example, if they meet certain requirements.

[0100] Preferably, each group contains no more than one region of interest from any given ultrasound image (i.e., each ultrasound image can contribute at most one region of interest to a group of ultrasound images).

[0101] In a particularly preferred example, where multiple ultrasound images comprise a sequence of ultrasound images, each group of regions of interest includes only regions of interest from sequentially adjacent ultrasound images. In other words, each region of interest (within a specific group of regions of interest) may come from an ultrasound image that is sequentially adjacent to an ultrasound image (in the sequence of ultrasound images) that contains another region of interest (specifically a group of regions of interest).

[0102] This approach means that a sequence of regions of interest (ROIs) can be efficiently constructed by comparing groups of ROIs from sequentially adjacent (i.e., immediately before or immediately after) ultrasound images. This produces groups of ROIs that are sequentially correlated (e.g., temporally correlated), providing additional information to aid in subsequent analysis of these groups.

[0103] The advantage of this method is that it provides a group of regions of interest that are sequential to each other, providing useful contextual information for subsequent analysis (e.g., because sequence information can provide important indicators of the presence of lesions), and it reduces the number of comparisons that need to be performed (because only regions of interest in sequentially adjacent ultrasound images need to be compared with each other).

[0104] In other words, before creating a group of regions of interest by grouping linked regions of interest into individual groups, step 330 can be performed by linking or associating regions of interest from sequentially adjacent ultrasound images together.

[0105] In the first case, step 330 may include determining a similarity measure between different regions of interest based on the relative overlap between the regions occupied by each region of interest in their respective ultrasound images. If this similarity measure exceeds a predetermined threshold, the regions of interest may be grouped together (or linked / associated with each other).

[0106] For example, the size of the overlap between the region occupied by one region of interest in an ultrasound image and the region occupied by another region in another ultrasound image can be used as a similarity measure. The size of the overlap can be determined using the intersection-over-union (IoU) metric. This process can establish a connection between regions of interest in different ultrasound images.

[0107] As an example only, consider a scenario where the region of interest in the first ultrasound image is defined by coordinates (X1, Y1, X2, Y). X The region of interest (IoU) in the second ultrasound image is defined by the bounding box (X3, Y3, X4, Y4). This coordinate information can be used to identify the relative overlap between the two IoU regions and can be used to calculate the IoU metric, as the coordinate information defines the coverage or extent of each IoU region.

[0108] In some examples, if the IoU metric is greater than a certain predetermined value, the regions of interest can be considered sufficiently similar (i.e., placed in the same group), the value preferably being no less than 0.4, for example, no less than 0.5, for example, no less than 0.6.

[0109] In the second scenario, step 330 includes grouping regions of interest based on metadata associated with each region of interest.

[0110] For example, the metadata can represent the features of a region of interest. Example metadata for a region of interest includes the location of the region of interest (e.g., the location of the center of the region of interest), the geometry (e.g., shape) and / or size of the region of interest, the region of interest (e.g., a value representing the confidence or probability that the region of interest contains a potential lesion—as can be provided by a machine learning method), and / or a feature map of the region of interest.

[0111] A feature map of a region of interest (ROI) is a map generated by applying one or more filters to the ROI, such as convolutional filters. More specifically, a feature map is a map generated by applying one or more layers of a neural network to the ROI. The meaning of the term "feature map" is easily understood in the field of machine learning.

[0112] In some examples, classifiers, such as machine learning methods, can process each region of interest to classify it into one of several categories. For instance, a set of regions of interest might only contain regions of interest that share the same category.

[0113] Preferably, a combination of the methods described above is used. In particular, for multiple images formed by a sequence of multiple images, each group of regions of interest may contain only the regions of interest from one or two other ultrasound images of one or more other regions of the group of regions of interest that are sequentially adjacent to each other, and these regions overlap with each other by more than a predetermined amount.

[0114] Method 300 includes step 340 of processing a group of regions of interest using machine learning methods to generate a prediction indicator that indicates the likelihood that the group of regions of interest contains a lesion in an individual.

[0115] In a particularly preferred example, the output of the machine learning method is data that changes in response to the predicted probability that a region group contains a lesion, for example in the form of a binary indicator, a categorical indicator, and / or a numerical indicator.

[0116] This process utilizes multiple similar regions of interest (ROIs) to predict whether they contain or depict lesions in an individual. This provides additional, context-sensitive information for generating predictive indicators, compared to simply processing a single ultrasound image. Because of the added contextual information, this method can provide a more accurate mechanism for determining the likelihood of lesion presence (within each group of ROIs).

[0117] By using a set of regions of interest, the proposed method also avoids the need to process full-size ultrasound images, which would be computationally expensive and / or require a lot of additional storage space.

[0118] It should also be noted that the proposed method improves the accuracy of identifying individual lesions because it uses a two-step process to identify or predict potential lesions.

[0119] Step 340 may include, for each group of regions of interest, processing the regions of interest to generate a sequence of regions of interest. This sequence may be a time series, such that the regions of interest are ordered based on the temporal order in which the original ultrasound images (for each region of interest) were initially captured. The region sequence can then be processed using machine learning methods to generate predictive indicators.

[0120] If obtained from a sequence of ultrasound images, the order of the sequence of the region of interest can be matched with the order of the sequence of ultrasound images from which the region of interest was obtained.

[0121] The sequence of the region of interest can be referred to as a “tube” or “tubelet”, and effectively represents a sequence (of a portion of the image) depicting the same underlying lesion. For example, a “tube” can effectively represent a video of an underlying lesion whose size / shape is suitable for viewing around the underlying lesion (rather than a video of the entire area imaged by the ultrasound probe).

[0122] In some examples, the sequence of regions of interest (ROIs) can include a stack of ROIs within a group. For example, if each ROI is a two-dimensional image, the sequence of ROIs can include a 3D volume representing a simple stack of two-dimensional images on top of each other.

[0123] In other words, a group of regions of interest can be effectively viewed as a "volume" that can be classified or processed using machine learning methods. That is, a sequence of regions of interest can be a single data structure containing stacked regions of interest, i.e., a combination of all regions of interest.

[0124] The input to machine learning methods can be the group of regions themselves, a sequence of regions of interest, and / or other data derived from the group of regions of interest, for example, by processing the regions of interest (e.g., using one or more filters or neural network layers).

[0125] A machine learning algorithm is any self-trained algorithm that processes input data to generate or predict output data. For the purposes of step 340, the input data includes data derived from a group of regions of interest, and the output data includes a predictive indicator indicating the likelihood that the group of regions of interest contains lesions in an individual.

[0126] Suitable machine learning algorithms for use in this invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks (e.g., CNN, RNNS, and / or LSTM). Other machine learning algorithms such as logistic regression, support vector machines, or Naive Bayes models are suitable alternatives.

[0127] Artificial neural networks (or simply neural networks) are inspired by the human brain. A neural network consists of layers, each containing multiple neurons. Each neuron performs a mathematical operation. Specifically, each neuron can include different weighted combinations of a single type of transformation (e.g., the same type of transformation, sigmoid, etc., but with different weights). In processing input data, each neuron performs mathematical operations on the input data, producing a numerical output. The outputs of each layer in the neural network are sequentially fed into the next layer. The final layer provides the output.

[0128] Methods for training machine learning algorithms are well-known. Typically, such methods involve obtaining a training dataset, which includes training input data entries and corresponding training output data entries (often labeled "ground truth" data). An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and its corresponding training output data entry is used to modify the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is often referred to as supervised learning.

[0129] For example, in machine learning algorithms formed by neural networks, the mathematical operations (weights) of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation, and others.

[0130] Training input data entries correspond to example data derived from a group of regions of interest (e.g., the group itself). Training output data entries correspond to example predictions (in binary, categorical, or numerical form) about the probability of lesions being present in the group.

[0131] Method 300 may further include step 350 of displaying a visually perceptible output at a display device in response to a prediction indicator.

[0132] Therefore, step 350 may include controlling a display device to provide visually perceptible output that responds to a prediction indicator (for each group of regions), indicating the predicted probability of the presence of a lesion in an individual. This indicator may, for example, provide a predicted probability of the presence of a lesion in the individual and / or a binary indicator (e.g., indicating "lesion present" or "lesion absent").

[0133] As another example, if the prediction indicator is a numerical value indicating the predicted probability, and one of the prediction indicators has a value exceeding a predetermined value, then step 350 may include controlling the display device to indicate the presence of a lesion.

[0134] In some examples, the visually perceptible output indicates which group (if any) of the regions of interest is predicted to contain or depict a lesion. This can be in the form of probabilities and / or binary indicators, for example, for each group of regions, or only for those regions that are considered likely to contain a lesion (e.g., with a probability exceeding a certain predetermined threshold).

[0135] In some examples, in response to determining that at least one group of regions of interest contains a lesion, for example, if the predicted probability exceeds a certain threshold, step 350 may include identifying the location of the identified lesion relative to the displayed ultrasound image of the patient; and in response to the identified location of the lesion, overlaying visually perceptible output (e.g., annotations, boxes, etc.) onto the displayed ultrasound image. The location of the identified lesion may be the location of a region of interest within the group of regions of interest predicted to contain a lesion in the displayed ultrasound image. In particular, visually perceptible output may be overlaid on the lesion location in the ultrasound image, for example at the location of the predicted region of interest containing the lesion. The ultrasound image may be one of multiple ultrasound images containing regions of interest that form part of a group of regions of interest in which lesions have been identified.

[0136] In some examples, all regions of interest are identified (e.g., using their respective labels), where regions of interest within a group of regions of interest that are predicted to contain lesions (e.g., with a probability greater than a predetermined value) are highlighted or otherwise emphasized, such as by a specific color, pattern, etc. This facilitates easier identification of potential areas where lesions may be found (e.g., for diagnostic purposes), while also providing information on the automatic prediction of the likelihood that a region contains a lesion.

[0137] In some examples, shading or transparency adjustment techniques are used to display information about the predicted likelihood that the area contains a lesion of interest.

[0138] In some examples, the display device can be configured to sequentially display each of a plurality of ultrasound images (e.g., play ultrasound video). Step 350 may include, in response to determining that at least one group of regions of interest may contain lesions (e.g., by processing a prediction indicator), controlling the display device to provide a visually perceptible output regarding the location of the regions of interest (from the group of regions of interest) associated with the currently displayed ultrasound image. In this way, the relative location of lesions associated with the group of regions of interest can be tracked or displayed over time.

[0139] Figure 2 and Figure 3 For the purpose of describing a relatively simple embodiment, each of the multiple ultrasound images includes an image-space ultrasound image.

[0140] However, as mentioned earlier, in some examples, each of the multiple ultrasound images may include a feature space ultrasound image, i.e., a feature space image. A feature space image may be a “conventional” ultrasound image that has undergone further processing (“image space image”), for example, by using one or more filters and / or layers of a neural network to generate the feature space image.

[0141] In some examples, these feature space images are processed in the same way as the previously described ultrasound images. However, in other examples, additional processing may be performed to identify (a set of) regions of interest.

[0142] Figure 4 The illustration shows the process of multiple ultrasound images including multiple feature space images 410, each feature space image being derived from a corresponding image space (ultrasound) image 420 (e.g., as a result of feature extraction process 430).

[0143] Therefore, the step of obtaining multiple ultrasound images may include obtaining multiple feature space ultrasound images. This can be performed by processing multiple image space ultrasound images to generate feature space ultrasound images in the feature extraction process 430.

[0144] The process of identifying regions of interest in ultrasound images may include independently processing the feature space ultrasound image to identify the region of interest within the feature space ultrasound image.

[0145] In an alternative embodiment, the process includes a process 435 of identifying regions of interest (ROIs) in the image-space ultrasound image 420. This can be performed using any of the previously described mechanisms, such as machine learning methods. ROIs in the feature-space ultrasound image can then be determined based on the ROIs in the image-space ultrasound image, as illustrated schematically using dashed lines. Specifically, the spatial relationships between different regions of the feature-space ultrasound image and the image-space ultrasound image are known and can therefore be used to identify relevant ROIs in the feature-space ultrasound image.

[0146] Therefore, the step of identifying regions of interest in ultrasound images may include identifying regions of interest based on regions of interest already identified in the original ultrasound images (used to derive multiple ultrasound images) before identifying regions of interest in multiple ultrasound images.

[0147] Then, in process 440, regions of interest in multiple (feature space) ultrasound images are grouped to generate one or more groups of regions of interest 460.

[0148] Each group of regions of interest 450 is processed using machine learning methods to generate predictive indicators 470 indicating the likelihood that the group of regions of interest contains a lesion. This can be performed using any of the methods previously described.

[0149] Figure 5 An exemplary process for grouping regions of interest is illustrated.

[0150] In particular, Figure 5The illustration shows a process for grouping regions of interest from consecutive ultrasound images within a sequence of ultrasound images (e.g., consecutive frames of an ultrasound video).

[0151] Figure 5 The illustration shows three ultrasound images, all two-dimensional and with the same resolution. Specifically, there are a first ultrasound image 510, a second ultrasound image 520, and a third ultrasound image 530. The ultrasound images form a (temporal) sequence of ultrasound images, such as frames representing an ultrasound video.

[0152] Each ultrasound image has been processed to identify regions of interest, for example, using machine learning methods. These regions of interest are... Figure 5 The diagram uses boxes to illustrate the point.

[0153] The regions of interest are then grouped. If the similarity metric between regions of interest (in adjacent frames) exceeds a predetermined threshold, this is done by linking or associating regions of interest in adjacent frames. The similarity metric can be the intersection-over-union (IoU) value, which is calculated by dividing the size of the overlap between regions (i.e., the size of the overlap if the regions are in the same ultrasound image) by the union of the regions.

[0154] Then, a sequence of regions of interest (ROIs) is obtained by selecting linked ROIs from different ultrasound images. Specifically, the obtained sequence of ROIs can be a sequence comprising at most one ROI from each ultrasound image, wherein each ROI in the sequence is linked or associated with another ROI in the sequence. The order of the ROI sequences can match the order of the sequences of the ultrasound images from which they were obtained.

[0155] Then you can delete the regions already placed in the region of interest (keeping the sequence within the region of interest) and repeat the process. Using Figure 5 The arrows in the diagram illustrate the process.

[0156] Figure 6 A processing system 600 according to one embodiment is illustrated. The processing system is illustrated in the context of the entire ultrasound imaging system 60.

[0157] The processing system 600 is configured to acquire multiple ultrasound images of an individual. These multiple ultrasound images may be acquired, for example, from a memory 610 and / or an ultrasound scanner 620 configured to generate ultrasound images.

[0158] The processing system 600 is also configured to identify regions of interest in ultrasound images, each region of interest being a portion of the ultrasound image that represents an area in an individual with a potential lesion.

[0159] The processing system 600 is also configured to group regions of interest from different ultrasound images together based on the similarity of the regions of interest; and to process each group of regions of interest using machine learning methods to predict whether the group of regions of interest contains a lesion in the individual.

[0160] The processing system 600 can also be configured to display visually perceptible output at a display device in response to each prediction indicator.

[0161] As previously stated, the processing system 600 can be suitably adapted to perform any of the methods described herein, with the necessary modifications, and those skilled in the art will be able to readily and suitably adapt the processing system 600.

[0162] Figure 7 The illustration shows the effectiveness of the proposed method in predicting whether an individual has one or more lesions.

[0163] First plot 710 illustrates the recall-precision curves (x-axis: recall, y-axis: precision) for correctly identifying lesions within two-dimensional ultrasound images using known two-dimensional neural network analysis methods. This plot was generated using benchmark data. Specifically, first plot 710 illustrates the recall-precision curves produced by using the Faster R-CNN lesion detection algorithm.

[0164] The second graph 720 illustrates the recall accuracy curve for correctly identifying lesions within multiple ultrasound images by performing the method disclosed herein. The second graph 720 is generated using the same baseline data used to generate the first graph 710.

[0165] The first graph 710 and the second graph 720 have the same scale (e.g., 0.0 to 1.0 on the x-axis and 0.0 to 1.0 on the y-axis).

[0166] Using the proposed method, higher precision values ​​can be obtained for higher levels of recall, and vice versa. In other words, the improvement of our invention shows that using additional context provided in multiple ultrasound images (e.g., utilizing ultrasound video) in the disclosed manner, which is not present in simple still images, can reduce false positive detections and / or missed detections of lesions.

[0167] Figure 8 This is a schematic diagram of a processing system 600 according to an embodiment of the present disclosure. As shown, the processing system 600 may include a (data) processor 860, a memory 864, and a communication module 868. These components may communicate directly or indirectly with each other, for example, via one or more buses.

[0168] Processor 860 may include a central processing unit (CPU), digital signal processor (DSP), ASIC, controller, FPGA, another hardware device, firmware device, or any combination thereof configured to perform the operations described herein. Processor 860 may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessor cores combined with a DSP, or any other such configuration. In some embodiments, the processor is a distributed processing system, for example, formed by a group of distributed processors.

[0169] Memory 864 may include cache memory (e.g., the cache memory of processor 860), random access memory (RAM), magnetoresistive RAM (MRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state storage devices, hard disk drives, other forms of volatile and non-volatile memory, or combinations of different types of memory. In one embodiment, memory 864 includes a non-transitory computer-readable medium. The non-transitory computer-readable medium may store instructions. For example, memory 864 or the non-transitory computer-readable medium may have program code recorded thereon, the program code including instructions for causing processor system 600 or one or more components of processor system 600, particularly processor 860, to perform the operations described herein. For example, processor system 600 may perform the operations of method 700. Instructions 866 may also be referred to as code or program code. The terms “instructions” and “code” should be interpreted broadly to include any type of computer-readable statement(s). For example, the terms "instruction" and "code" can refer to one or more programs, routines, subroutines, functions, procedures, etc. "Instruction" and "code" can include a single computer-readable statement or multiple computer-readable statements. A memory 864 containing code can be referred to as a computer program product.

[0170] The communication module 868 may include any electronic circuitry and / or logic circuitry to facilitate direct or indirect communication of data between the processing system 600, the penetration device, and / or the user interface (or other further devices). In this respect, the communication module 868 may be an input / output (I / O) device. In some cases, the communication module 868 facilitates direct communication between the processor circuitry 600 and / or the system 600. Figure 6 Direct or indirect communication between various components.

[0171] It should be understood that the disclosed method is preferably a computer-implemented method. Therefore, the concept of a computer program comprising computer program code is also proposed, which, when run on a processing system such as a computer or a set of distributed processors, is used to implement any described method.

[0172] Different portions, lines, or blocks of code of a computer program according to embodiments may be executed by a processing system or computer to perform any of the methods described herein. In some alternative implementations, functions marked in one or more block diagrams or flowcharts may not appear in the order indicated in the diagrams. For example, depending on the functions involved, two blocks shown successively may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order.

[0173] This disclosure discloses a computer program (product) including instructions that, when run by a computer or processing system, cause the computer or processing system to perform any of the steps of the methods described herein. The computer program (product) may be stored on a non-transitory computer-readable medium.

[0174] Similarly, a computer-readable (storage) medium including instructions, when executed by a computer or processing system, causes the computer or processing system to perform any of the methods (steps) described herein. A computer-readable data carrier having a previously described computer program (product) stored thereon is also proposed. A data carrier signal carrying a previously described computer program (product) is also proposed.

[0175] Those skilled in the art, through studying the accompanying drawings, the disclosure, and the claims, will be able to understand and implement variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. Although specific measures are recited in dissimilar dependent claims, this does not imply that combinations of these measures cannot be advantageously used.

[0176] The disclosed methods are preferably computer-implemented methods and can be executed by a suitable processing system. Any processing system described herein can be suitably adapted to perform any method described herein, just as any method described herein can be adapted to perform a process executed by any processing system described herein.

[0177] A computer program product including computer program code units is also provided, which, when run on a computing device having a processing system, cause the processing system to perform all the steps of any of the methods described herein.

[0178] A single processor or other unit can perform the functions of several items recounted in the claims. Computer programs can be stored / distributed on suitable media such as optical storage media or solid-state media provided with or as part of other hardware, but can also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems.

[0179] Although specific measures are described in different dependent claims, this does not imply that combinations of these measures cannot be used advantageously.

[0180] If the term "suitable" is used in the claims or description, it should be noted that the term "suitable" is intended to be equivalent to the term "configured as". Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A processing system (600) for predicting the likelihood of an individual having one or more lesions, the processing system comprising: Memory, and The processor is configured as follows: Obtain multiple ultrasound images (210, 410) of the individual (310), the multiple ultrasound images including a time series of ultrasound images; Identify (220, 320, 430, 435) regions of interest (230) in the plurality of ultrasound images, each region of interest being a portion of the ultrasound images representing a region with a potential lesion in the individual; Based on the similarity of the regions of interest, the regions of interest from different ultrasound images from the multiple ultrasound images are grouped together (240, 330, 440), wherein the grouping is performed without requiring any spatial registration or alignment between the different ultrasound images; and Each set of regions of interest (250, 450) is processed (340) using machine learning methods to generate prediction indicators (270, 470) that indicate the likelihood that the set of regions of interest contains a lesion in the individual.

2. The processing system (600) according to claim 1, wherein, Each ultrasound image is a two-dimensional ultrasound image.

3. The processing system (600) according to claim 1 or 2, wherein, The processor is configured to process each set of regions of interest by performing a procedure that includes the following operations for each set of regions of interest: The sequence of regions of interest is generated using the regions of interest from the group of regions of interest; and The sequence of the region is processed using machine learning methods to generate a predictive indicator that indicates the likelihood that the sequence of the region contains a lesion in the individual.

4. The processing system (600) according to claim 3, wherein, The processor is configured to generate a sequence of regions by performing steps including stacking the regions of interest.

5. The processing system (600) according to claim 1 or 2, wherein, The processor is configured to identify regions of interest in the ultrasound images by performing a process that includes processing each ultrasound image using a second machine learning method to identify regions of interest.

6. The processing system (600) according to claim 1, wherein, The multiple ultrasound images include videos of ultrasound images.

7. The processing system (600) according to claim 6, wherein, The order of the sequence of ultrasound images depends on the time at which each ultrasound image was captured.

8. The processing system (600) according to claim 6 or 7, wherein, Within each set of regions of interest, each region of interest is derived from ultrasound images sequentially adjacent to ultrasound images of another region of interest within the same set of regions of interest.

9. The processing system (600) according to claim 1 or 2, wherein, Each region of interest is no larger than 0.25 times the size of the ultrasound image.

10. The processing system (600) according to claim 1 or 2, wherein, The processor is configured to display (350) a visually perceptible output at a display device in response to each prediction indicator.

11. The processing system (600) according to claim 10, wherein, The processor is configured to: if at least one prediction indicator indicates that at least one set of regions of interest may contain lesions, then: Identify the position of the identified lesion relative to the displayed ultrasound image of the patient; and In response to the identified location of the lesion, the visually perceptible output is superimposed on the displayed ultrasound image.

12. The processing system (600) according to claim 1 or 2, wherein, The processor is configured to group regions of interest by performing a process that includes the following operations: A similarity metric is determined between different regions of interest (ROIs) from different ultrasound images. This similarity metric represents the size of the overlap between the relative area occupied by one ROI within an ultrasound image and the relative area occupied by another ROI in another ultrasound image. In response to the similarity metric between two different regions of interest exceeding a predetermined threshold, the two different regions of interest are grouped into the same group.

13. An ultrasound imaging system (60), comprising: An ultrasound scanner (620) is used to generate multiple ultrasound images (210, 410) of an individual, the multiple ultrasound images including a time series of ultrasound images; The processing system (600) according to any one of claims 1 to 11 for predicting the likelihood of an individual having one or more lesions based on the multiple ultrasound images.

14. A computer-implemented method (300) for predicting the likelihood that an individual has one or more lesions, the computer-implemented method comprising: Obtain multiple ultrasound images (210, 410) of the individual (310), the multiple ultrasound images including a time series of ultrasound images; Identify (320) regions of interest (230) in the plurality of ultrasound images, each region of interest being a portion of the ultrasound images representing a region with a potential lesion in the individual; Based on the similarity of the regions of interest, the regions of interest from different ultrasound images from the plurality of ultrasound images are grouped (330) together, wherein the grouping is performed without requiring any spatial registration or alignment between the different ultrasound images; and Each set of regions of interest (250, 450) is processed (340) using machine learning methods to generate prediction indicators (270, 470) that indicate the likelihood that the set of regions of interest contains the lesion in the individual.

15. A computer program product comprising computer program code units, which, when run on a computing device having a processing system, cause the processing system to perform all the steps of the computer-implemented method according to claim 14.

Images