Ultrasound machine learning techniques using transformed image data

Abstract

According to an embodiment, there is provided a method for analyzing ultrasound image data (300) obtained from ultrasound imaging, the method comprising: obtaining the ultrasound image data by an ultrasound probe (104); transforming, by a processor (132), the ultrasound image data with at least one transformation to generate at least one set of transformed data (320, 330, 340); inputting at least one set of the ultrasound image data and the transformed data into a machine learning model (400), wherein the machine learning model is implemented by the processor; implementing, by the processor, the machine learning model using at least one set of the ultrasound image data and the transformed data; and identifying, by the processor, at least one feature in the ultrasound image data as determined by the machine learning model.

Description

Technical Field

[0001] Certain embodiments relate to ultrasonic imaging. More specifically, certain embodiments relate to techniques for determining one or more features in an ultrasonic image by using transformed image data. Background Art

[0002] Ultrasonic imaging is a medical imaging technique for imaging the human anatomy. Ultrasonic imaging can be used to image or analyze the blood flow through a patient's cardiovascular system. Ultrasonic imaging uses real-time, non-invasive high-frequency sound waves to generate two-dimensional (2D), three-dimensional (3D), and / or four-dimensional (4D) (i.e., real-time / continuous 3D image data) image data.

[0003] An ultrasonic system obtains image data. For example, the image data can be B-mode data, which can display the reflection intensities at different spatial positions within a patient (hereinafter referred to as a spatial B-mode image or image data). A machine learning model can use this data to identify features in the ultrasonic image. Such features include the patient's organs, including diseased organs. Examples of such organs include the liver, kidney, pancreas, or spleen. Techniques for improving the ability of a machine learning model to identify features such as a patient's organ with a malignant lesion may be helpful.

[0004] By comparing such systems with some aspects of the present disclosure set forth in the accompanying drawings in the remainder of the present application, more limitations and disadvantages of conventional and traditional methods will become apparent to those skilled in the art. Summary of the Invention

[0005] According to an embodiment, a method for analyzing ultrasound image data obtained from ultrasound imaging is provided. The method includes: obtaining the ultrasound image data by an ultrasound probe; transforming, by a processor, the ultrasound image data using at least one transform to generate at least one set of transformed data; inputting the ultrasound image data and the at least one set of transformed data into a machine learning model, wherein the machine learning model is implemented by the processor; implementing, by the processor, the machine learning model using the ultrasound image data and the at least one set of transformed data; and identifying, by the processor, at least one feature in the ultrasound image data as determined by the machine learning model. The method may further include: determining a range of the ultrasound image data according to a region of interest. The machine learning model may include a convolutional neural network. The method may further include training the machine learning model by: inputting annotated ultrasound image data into the machine learning model, wherein the ultrasound image data indicates the presence or absence of the at least one feature; using at least one transform to transform the ultrasound image data to generate transformed data; and inputting the transformed data into the machine learning model. The method may further include: updating the machine learning model to reduce a loss function when the machine learning model receives additional ultrasound image data indicating the presence or absence of the at least one feature. The ultrasound image data may include spatial B-mode image data. The at least one set of transformed data may include at least one of Fourier transform data, shear transform data, or Hadamard transform data. The at least one set of transformed data may include only one of Fourier transform data, shear transform data, or Hadamard transform data. The at least one set of transformed data may include only two of Fourier transform data, shear transform data, or Hadamard transform data. The at least one set of transformed data may include Fourier transform data, shear transform data, and Hadamard transform data. The at least one feature may include at least one of a patient's organ not having a lesion or having a lesion (benign or malignant).

[0006] According to an embodiment, a system for analyzing ultrasound image data obtained from ultrasound imaging includes: an ultrasound probe configured to obtain the ultrasound image data; and a processor configured to: transform the ultrasound image data using at least one transform to generate at least one set of transformed data; input the ultrasound image data and the at least one set of transformed data into a machine learning model, wherein the machine learning model is implemented by the processor; implement the machine learning model using the ultrasound image data and the at least one set of transformed data; and identify at least one feature in the ultrasound image data as determined by the machine learning model. The processor may be further configured to determine a range of the ultrasound image data based on a region of interest. The machine learning model may include a convolutional neural network. The processor may be configured to train the machine learning model by: inputting annotated ultrasound image data into the machine learning model, wherein the ultrasound image data indicates the presence or absence of the at least one feature; using at least one transform to transform the ultrasound image data to generate transformed data; and inputting the transformed data into the machine learning model. The processor may be configured to implement the machine learning model to reduce a loss function when the machine learning model receives additional ultrasound image data indicating the presence or absence of the at least one feature. The ultrasound image data may include spatial B-mode image data. The at least one set of transformed data may include at least one of Fourier transform data, shear transform data, or Hadamard transform data. The at least one set of transformed data may include only one of Fourier transform data, shear transform data, or Hadamard transform data. The at least one feature may include at least one of a patient's organ being normal or having a lesion (e.g., benign or malignant).

[0007] These and other advantages, aspects, and novel features of the present disclosure, as well as details of its illustrative embodiments, will be more fully understood from the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram of an exemplary ultrasound system according to various embodiments, the exemplary ultrasound system being operable to identify features in ultrasound image data using a machine learning model.

[0009] Figure 2 is an exemplary spatial B-mode image and a region of interest (or ROI) therein.

[0010] Figure 3A shows spatial B-mode image data in a region of interest.

[0011] Figures 3B to 3D shows transforming spatial B-mode image data into different domains according to various embodiments.

[0012] Figure 4 A block diagram showing a machine learning model that trains or uses processed spatial B-mode image data or at least one other transformed data according to various embodiments.

[0013] Figure 5 A flowchart illustrating exemplary steps according to various embodiments that can be used to train a machine learning model to identify one or more types of features in a spatial B-mode image.

[0014] Figure 6 A flowchart illustrating exemplary steps according to various embodiments that can be used to use a machine learning model to identify one or more types of features in a spatial B-mode image.

[0015] Figure 7 A representation of a machine learning model for identifying one or more types of features in a spatial B-mode image according to various embodiments. DETAILED DESCRIPTION

[0016] Certain embodiments may be present in methods and systems for identifying one or more features in spatial B-mode image data or other image data obtained by an ultrasound system. Such feature identification uses a trained machine learning model. Such a machine learning model (or more simply, a model) may receive spatial B-mode image data and one or more transformed data as inputs. When training the model, the same type of data (spatial B-mode image data and one or more transformed data) may be input into the model.

[0017] Aspects of the present disclosure have the technical effect of using machine learning to enhance the identification of features in ultrasound images to assist in providing a diagnosis. Various embodiments have the technical effect of using machine learning to process the acquired ultrasound image data to identify features. Certain embodiments have the technical effect of determining the probability of the presence of certain features in the ultrasound image data. Aspects of the present disclosure have the technical effect of improving the identification of one or more features in ultrasound images. Aspects of the present disclosure have the technical effect of using transformed data to identify features in ultrasound images.

[0018] The foregoing Summary of the Invention and the following detailed description of certain embodiments will be better understood when read in conjunction with the accompanying drawings. To the extent that the drawings illustrate diagrams of functional blocks of various embodiments, these functional blocks do not necessarily represent divisions between hardware circuitry. Thus, for example, one or more functional blocks (e.g., a processor or a memory) may be implemented in a single piece of hardware (e.g., a general-purpose signal processor or a random access memory block, a hard disk, etc.) or in multiple pieces of hardware. Similarly, a program may be a stand-alone program, may be incorporated as a subroutine into an operating system, may be a function in an installed software package, etc. It should be understood that the various embodiments are not limited to the arrangements and instrumentalities shown in the drawings. It should also be understood that embodiments may be combined, or other embodiments may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the various embodiments. Accordingly, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0019] As used herein, an element or step recited in the singular and preceded with the word "a" or "an" should be understood as not excluding a plurality of the recited elements or steps, unless expressly stated to the contrary. Further, references to "exemplary embodiments", "various embodiments", "certain embodiments", "representative embodiments", etc. are not to be construed as precluding the existence of additional embodiments that also incorporate the recited features. Moreover, unless expressly stated to the contrary, an embodiment that "comprises", "includes", or "has" one or more elements having a particular attribute may include additional elements that do not have that attribute.

[0020] Moreover, as used herein, the term "image" broadly refers to both visual images and data representing visual images (image data). However, many embodiments generate (or are configured to generate) at least one visual image. Additionally, as used herein, the phrase "image" is used to refer to an ultrasound mode, which may be one-dimensional (1D), two-dimensional (2D), three-dimensional (3D), or four-dimensional (4D), and includes brightness mode (B-mode or also referred to as spatial B-mode), motion mode (M-mode), color motion mode (CM-mode), color flow mode (CF-mode), pulsed wave (PW) Doppler, continuous wave (CW) Doppler, contrast-enhanced ultrasound (CEUS), and / or sub-modes of B-mode and / or CF-mode, such as harmonic imaging, shear wave elastography imaging (SWEI), strain elastography, tissue velocity imaging (TVI), power Doppler imaging (PDI), B-flow, microvascular imaging (MVI), ultrasound-guided attenuation parameter (UGAP), etc.

[0021] In addition, as used herein, the term processor or processing unit refers to any type of processing unit that can perform the required calculations needed for the various embodiments, such as a single-core or multi-core CPU, an accelerated processing unit (APU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination thereof. The processor or processing unit may include multiple processors located in the same location (e.g., integrated together in a single ASIC) or distributed in different locations. When there are multiple processors, these processors may communicate with and / or work together with other associated processors to achieve processing and calculations.

[0022] It should be noted that the various embodiments described herein for generating or forming an image may include processing for forming the image, which in some embodiments includes beamforming, and in other embodiments does not include beamforming. For example, an image may be formed without beamforming, such as by multiplying a matrix of demodulated data by a coefficient matrix such that the product is the image, and where the process does not form any "beams". Additionally, the formation of the image may be performed using channel combinations (e.g., synthetic aperture techniques) that may originate from more than one transmit event.

[0023] In various embodiments, for example, ultrasonic processing is performed in software, firmware, hardware, or a combination thereof to form an image, including ultrasonic beamforming, such as receive beamforming. A specific implementation of an ultrasonic system having a software beamformer architecture formed according to various embodiments is shown in Figure 1 as follows.

[0024] Figure 1 is a block diagram of an exemplary ultrasonic system operable to identify features in image data obtained from a patient. Referring to Figure 1 , an ultrasonic system 100 and a training system 200 are shown. The ultrasonic system 100 includes a transmitter 102, an ultrasonic probe 104, a transmit beamformer 110, a receiver 118, a receive beamformer 120, an analog-to-digital (A / D) converter 122, a radio frequency (RF) processor 124, an RF quadrature (RF / IQ) buffer 126, a user input device 130, a signal processor 132, an image buffer 136, a display system 134, and an archive 138.

[0025] The transmitter 102 may include suitable logic, circuitry, interfaces, and / or code that is operable to drive the ultrasound probe 104. The ultrasound probe 104 may be a linear, convex, intracavitary, or phased array transducer. The ultrasound probe 104 may include a two-dimensional (2D) array of piezoelectric elements. The ultrasound probe 104 may include a set of transmit transducer elements 106 and a set of receive transducer elements 108 that generally comprise the same elements. The set of transmit transducer elements 106 may emit ultrasound signals that pass through the oil and the probe cap and into the target. In a representative embodiment, the ultrasound probe 104 may be operable to acquire ultrasound image data that covers at least a substantial portion of an anatomical structure (such as the liver, kidney, pancreas, spleen, kidney, or any suitable anatomical structure). In an exemplary embodiment, the ultrasound probe 104 may operate in a volume acquisition mode, in which the transducer assembly of the ultrasound probe 104 acquires a plurality of parallel 2D ultrasound slices that form an ultrasound volume.

[0026] The transmit beamformer 110 may include suitable logic, circuitry, interfaces, and / or code that is operable to control the transmitter 102, which drives the set of transmit transducer elements 106 via a transmit sub-aperture beamformer 114 to emit an ultrasound transmit signal into the region of interest (e.g., a person, an animal, a subsurface cavity, a physical structure, etc.). The emitted ultrasound signal may be backscattered from structures (such as blood cells or tissue) in the object of interest to produce echoes. The echoes are received by the receive transducer elements 108.

[0027] The set of receive transducer elements 108 in the ultrasound probe 104 may be operable to convert the received echoes into analog signals, perform sub-aperture beamforming via a receive sub-aperture beamformer 116, and then transmit them to the receiver 118. The receiver 118 may include suitable logic, circuitry, interfaces, and / or code that is operable to receive the signals from the receive sub-aperture beamformer 116. The analog signals may be communicated to one or more of a plurality of A / D converters 122.

[0028] The plurality of A / D converters 122 may include suitable logic, circuitry, and interfaces and / or code that is operable to convert the analog signals from the receiver 118 into corresponding digital signals. The plurality of A / D converters 122 are disposed between the receiver 118 and the RF processor 124. However, the present disclosure is not limited in this regard. Thus, in some embodiments, the plurality of A / D converters 122 may be integrated within the receiver 118.

[0029] The RF processor 124 may include suitable logic, circuitry, interfaces, and / or code that is operable to demodulate digital signals output by the plurality of A / D converters 122. According to an embodiment, the RF processor 124 may include a complex demodulator (not shown) that is operable to demodulate the digital signals to form an I / Q data pair representing a corresponding echo signal. The RF or I / Q signal data may then be communicated to the RF / IQ buffer 126. The RF / IQ buffer 126 may include suitable logic, circuitry, interfaces, and / or code that is operable to provide temporary storage of the RF or I / Q signal data generated by the RF processor 124.

[0030] The receive beamformer 120 may include suitable logic, circuitry, interfaces, and / or code that is operable to perform digital beamforming processing to, for example, sum the delayed channel signals received from the RF processor 124 via the RF / IQ buffer 126 and output a beam sum signal. The resulting processed information may be the beam sum signal that is output from the receive beamformer 120 and communicated to the signal processor 132. According to some embodiments, the receiver 118, the plurality of A / D converters 122, the RF processor 124, and the beamformer 120 may be integrated into a single beamformer, which may be a digital beamformer. In various embodiments, the ultrasound system 100 includes a plurality of receive beamformers 120.

[0031] The user input device 130 can be used to input patient data, scan parameters, settings, select protocols and / or templates, select target structures for image acquisition, input and / or select regions of interest, modify regions of interest, select regions of interest for image acquisition, focus / zoom volumes, etc. In an exemplary embodiment, the user input device 130 can be operable to configure, manage, and / or control the operation of one or more components and / or modules in the ultrasound system 100. In this regard, the user input device 130 can be operable to configure, manage, and / or control the operation of the transmitter 102, ultrasound probe 104, transmit beamformer 110, receiver 118, receive beamformer 120, RF processor 124, RF / IQ buffer 126, user input device 130, signal processor 132, image buffer 136, display system 134, and / or archive 138. The user input device 130 can include buttons, rotary encoders, touchscreens, motion tracking, voice recognition, mouse devices, keyboards, cameras, and / or any other device capable of receiving user instructions. In certain embodiments, for example, one or more user input devices in the user input device 130 can be integrated into other components such as the display system 134 or the ultrasound probe 104. For example, the user input device 130 can include a touchscreen display.

[0032] The signal processor 132 can include suitable logic, circuitry, interfaces, and / or code that can be operable to process ultrasound scan data (e.g., summed IQ signals) to generate ultrasound images for presentation on the display system 134. The signal processor 132 can be operable to perform one or more processing operations according to multiple ultrasound modalities of the acquired ultrasound scan data, such as B-mode, Doppler modality, and color Doppler modality. In an exemplary embodiment, the signal processor 132 can be operable to perform display processing and / or control processing, etc. When an echo signal is received, the acquired ultrasound scan data, such as spatial B-mode data, can be processed in real time during a scan session. Additionally or alternatively, the ultrasound scan data can be temporarily stored in the RF / IQ buffer 126 during a scan session and processed in a less real-time manner in online or offline operations. In various embodiments, the processed image data can be presented at the display system 134 and / or stored at the archive 138. The archive 138 can be a local archive, a Picture Archiving and Communication System (PACS), or any suitable device for storing images and related information.

[0033] The signal processor 132 can be one or more central processing units, microprocessors, microcontrollers, etc. For example, the signal processor 132 can be an integrated component or can be distributed at various locations. In an exemplary embodiment, the signal processor 132 can include a data transformation processor 140 (or an image data transformation processor 140) and a feature recognition processor 150. The signal processor 132 can be capable of receiving input information from the user input device 130 and / or the file 138, generating an output that can be displayed by the display system 134, and manipulating the output in response to the input information from the user input device 130, etc. The signal processor 132, the data transformation processor 140, and / or the feature recognition processor 150 can be capable of executing any of the methods and / or instruction sets discussed herein, for example, according to the various embodiments.

[0034] The ultrasound system 100 is operable to continuously acquire ultrasound scan data at a frame rate suitable for the imaging situation under consideration. Typical frame rates are in the range of 20 to 120 per second, but can be lower or higher. As used herein, "time" or "time period" can correspond to one or more frames. The acquired ultrasound scan data can be displayed on the display system 134 at the same frame rate or at a display rate slower or faster than the frame rate. A series of images (such as of a patient's blood flow) can be displayed simultaneously. An image buffer 136 is included for storing frames of the acquired ultrasound scan data that are not scheduled for immediate display. Preferably, the image buffer 136 has sufficient capacity to store ultrasound scan data frames equivalent to at least several minutes. The frames of the ultrasound scan data are stored in a manner that is readily retrievable according to their acquisition order or time. The image buffer 136 can be embodied as any known data storage medium.

[0035] The signal processor 132 can include a data transformation 140 that includes suitable logic, circuitry, interfaces, and / or code that is operable to transform ultrasound image data using the ultrasound probe 104. In an exemplary embodiment, the data transformation processor 140 can be configured to receive image data (such as, for example, spatial B-mode image data or a portion thereof, such as data in a region of interest) and transform the image data using one or more transforms (such as, for example, a Fourier transform, a slant transform, and / or a Hadamard transform or Walsh transform). The data transformation processor 140 can be configured to receive user input selecting a region of interest prior to performing an ultrasound image acquisition and analyzing the ultrasound image data and / or volume of the ultrasound image acquisition to obtain a series of images over time. The data transformation processor 140 can transform the received image data for each image or the selected images in the image.

[0036] The display system 134 can be any device capable of conveying visual information to a user. For example, the display system 134 can include a liquid crystal display, a light emitting diode display, and / or any suitable one or more displays. The display system 134 is operable to present 2D ultrasound images, 2D sequential ultrasound images, biplane ultrasound images, biplane ultrasound slices extracted from 3D / 4D volumes, rendered 3D / 4D volumes, selectable target structures, and / or any suitable information.

[0037] The archive 138 can be one or more computer-readable memories integrated with and / or communicatively coupled to the ultrasound system 100 (e.g., via a network), such as a picture archiving and communication system (PACS), a server, a hard disk, a floppy disk, a CD, a CD-ROM, a DVD, a compact storage device, a flash memory, a random access memory, a read-only memory, an electrically erasable and programmable read-only memory, and / or any suitable memory. The archive 138 can include, for example, a database, a library, an information set, or other memory accessed by and / or in conjunction with the signal processor 132. For example, the archive 138 can be capable of storing data temporarily or permanently. The archive 138 can be capable of storing medical image data, data generated by the signal processor 132, and / or instructions readable by the signal processor 132, etc. In various embodiments, for example, the archive 138 stores 2D ultrasound images, 2D sequential ultrasound images, biplane ultrasound images, biplane ultrasound slices extracted from 3D / 4D volumes, rendered 3D / 4D volumes, instructions for acquiring ultrasound image data, instructions for generating sequential ultrasound images, instructions for generating sample sequential ultrasound images, instructions for classifying an image as a generated image or a real image, instructions for providing feedback based on the classification of the image, instructions for determining that a target function has been reached, instructions for generating enhanced sequential ultrasound images.

[0038] The components of the ultrasound system 100 can be implemented in software, hardware, firmware, etc. The various components of the ultrasound system 100 can be communicatively connected. The components of the ultrasound system 100 can be implemented separately and / or integrated in various forms. For example, the display system 134 and the user input device 130 can be integrated as a touch screen display.

[0039] Still referring to Figure 1, the training system 200 may include a training engine 210 and a training database 220. The training engine 210 may include suitable logic, circuitry, interfaces, and / or code that are operable to train the neurons of a deep neural network (e.g., an artificial intelligence model) inferred (i.e., deployed) by the data transformation processor 140 and / or the feature recognition processor 150. For example, the machine learning model implemented by the feature recognition processor 150 may be trained to identify features in the spatial image data obtained by the ultrasound system 100.

[0040] In various embodiments, the database 220 of training images may be a Picture Archiving and Communication System (PACS) or any suitable data storage medium. In certain embodiments, the training engine 210 and / or the training image database 220 may be a remote system communicatively coupled to the ultrasound system 100 via a wired or wireless connection, as Figure 1 shown. Additionally and / or alternatively, some or all of the components of the training system 200 may be integrated with the ultrasound system 100 in various forms. In some examples, the training image database may include reference sequence ultrasound images of anatomical structures and / or tissues. In some examples, the reference sequence ultrasound images may be generated by the sequence image acquisition processor 140 and provided to the training image database 220.

[0041] Figure 2 are exemplary spatial B-mode image data 300 and a region of interest 301 therein. The spatial B-mode image data 300 shows an image including a liver region. The spatial B-mode image data 300 may be presented on the display system 134 for a user to view. The region of interest 301 defines a subset 310 of the spatial B-mode image data 300 and may be drawn and / or located by the user in the ultrasound image data 300 according to clinical purposes. The user may draw and / or locate the region of interest 301 via the user input device 130. As generally disclosed herein, the spatial B-mode image data 300 is used as an example, but other types of image data may be applied according to the techniques described herein. Such other types of image data include Doppler image data or color Doppler image data. In a multimodal (e.g., capable of obtaining spatial B-mode image data and Doppler image data) system, multiple types of image data may be used in addition to the transformed data discussed further below.

[0042] Referring again to Figure 1, the data transformation processor 140 can be configured to collect ultrasound image data when the ultrasound probe 104 slides over an area of interest (e.g., area of interest 310), anatomical structure, tissue, and / or fluid contained therein (such as blood flowing through the area of interest in the patient's cardiovascular system). When the ultrasound probe 104 slides over such an area, the data transformation processor 140 collects ultrasound images and transforms the image data according to one or more techniques. The data provided to the data transformation processor 140 can be stored in the archive 138 and / or any suitable computer-readable medium, and the data transformation processor 140 can obtain ultrasound image data from the archive 138 and / or any suitable computer-readable medium. The data transformation processor 140 can generate Figure 3B , Figure 3C and Figure 3D the images shown. Figure 3A shows a subset of the spatial B-mode image data 310 from Figure 2 .

[0043] Figure 3B shows the Fourier transform data 320, which is the result of transforming a subset of the spatial B-mode image data corresponding to the area of interest 310 using the Fourier transform and in particular the discrete Fourier transform. Figure 3B The specific Fourier transform data 320 shown is a graphical representation of the transformation of a subset of the spatial B-mode image data 310 using the Fourier transform. In this case, the Fourier transform represents the image as a sum of complex exponentials of different amplitudes, frequencies, and phases. The two-dimensional discrete Fourier transform can be described by the following equation:

[0044]

[0045] where K1 is N1 - 1, K2 is N2 - 1, and N1 and N2 are integers corresponding to the size of the input image.

[0046] The data transformation processor 140 may be able to perform one or more such Fourier transforms on one or more types of input image data. For example, the data transformation processor 140 can use two Fourier transforms to transform the image data 310 to generate a set of two Fourier transform data 320, and each of the Fourier transform data in the Fourier transform data 320 can be associated with Figure 3BThe single Fourier transform data 320 depicted therein is used in the same manner and is used as an example herein. As another example, the data transformation processor 140 may use multiple inputs (such as spatial B-mode image data and Doppler image data) to generate multiple Fourier transform data 320. Image data from other modalities (e.g., color Doppler image data) may be included as part of this technique. Additionally, multiple Fourier transforms may be used for any given set of image data received by the data transformation processor 140. For example, the data transformation processor 140 may use two types of Fourier transforms on the spatial B-mode image data and two types of Fourier transforms on the Doppler image data, resulting in a set of four transformed data. Image data obtained using different modalities may still correspond to the same region of interest 310.

[0047] Figure 3C Shown is the shear transform data 330, which is the result of transforming a subset of the spatial B-mode image data corresponding to the region of interest 310 using a shear transform. Figure 3C The particular shear transform data 330 shown is a graphical representation of the result of transforming a subset of the spatial B-mode image data 310 using a shear transform. The shear transform can be described as follows. As the name implies, the first sequence vector of the shear transform has a "tilted" or stepped waveform. The matrix of the shear transform can be formed by multiplying a series of sparse matrices on a Hadamard matrix. This will be shown as follows. In the following example, let S * N represent the shear matrix of order N. The asterisk is used to indicate that the order of the sequence is the same as the Hadamard matrix of the same order N. The subscript indicates the order of the matrix. Where N = 2, S * N is given by:

[0048]

[0049] In the case of N = 4, S * N is given by:

[0050]

[0051] The data transformation processor 140 may be capable of performing one or more such shear transforms on one or more types of input image data. For example, the data transformation processor 140 may use two shear transforms to transform the image data 310 to generate a set of two shear transform data 330, and each of the shear transform data 330 may be related to that in Figure 3CThe single shear transform data 330 depicted is used in the same manner and is used herein as an example. As another example, the data transformation processor 140 can use multiple inputs (such as spatial B-mode image data and Doppler image data) to generate multiple shear transform data 330. Image data from other modalities (e.g., color Doppler image data) can be included as part of this technique. Additionally, multiple shear transforms can be used for any given set of image data received by the data transformation processor 140. For example, the data transformation processor 140 can use two types of shear transforms for the spatial B-mode image data and two types of shear transforms for the Doppler image data, resulting in a set of four transformed data. Image data obtained using different modalities can still correspond to the same region of interest 310.

[0052] Figure 3D The Hadamard transform data 340 is shown, which is the result of transforming a subset of the spatial B-mode image data corresponding to the region of interest 310 using the Hadamard transform. Figure 3D The specific Hadamard transform data 340 shown is a graphical representation of the result of transforming a subset of the spatial B-mode image data 310 using the Hadamard transform.

[0053] Hadamard transform H m is 2 m ×2 m matrix, which transforms 2 m real numbers x n into 2 real number elements X k . The Hadamard transform can be defined in two ways: recursively or by using the binary (base 2) representation of the indices n and k. Recursively, the 1×1 Hadamard transform H0 is defined by the identity H0 = 1, and then H m is defined for m > 0 by:

[0054]

[0055] where the 1 on the square root of 2 is an optional normalization factor.

[0056] For M > 1, H m can also be defined as:

[0057]

[0058] where

[0059]

[0060] denotes the Kronecker product

[0061] Thus, except for the normalization factor, a Hadamard matrix can consist entirely of 1s and -1s. Additionally, a Hadamard matrix can be defined by its (k,n)th entry as follows:

[0062]

[0063] where k j and n j are bit elements (0 or 1) or k and n, respectively.

[0064] For the top left element, k = n = 0. This results in:

[0065]

[0066] The data transformation processor 140 can be capable of performing one or more such Hadamard transforms on one or more types of input image data. For example, the data transformation processor 140 can use two Hadamard transforms to transform the image data 310 to generate two sets of Hadamard transform data 340, and each of the Hadamard transform data 340 in the Hadamard transform data 340 can be used in the same manner as the single Hadamard transform data 340 depicted in Figure 3D and is used as an example herein. As another example, the data transformation processor 140 can use multiple inputs (such as spatial B-mode image data and Doppler image data) to generate multiple sets of Hadamard transform data 340. Image data from other modalities (e.g., color Doppler image data) can be included as part of this technique. Additionally, multiple Hadamard transforms can be used for any given set of image data received by the data transformation processor 140. For example, the data transformation processor 140 can use two types of Hadamard transforms on the spatial B-mode image data and two types of Hadamard transforms on the Doppler image data, resulting in four sets of transformed data. Image data obtained using different modalities can still correspond to the same region of interest 310.

[0067] Although the embodiments herein describe Fourier transforms, slant transforms, and Hadamard transforms, other types of transforms can be used as part of or in combination with the techniques described herein. Such transforms can also include the Hartley transform.

[0068] The feature recognition processor 150 may include suitable logic, circuitry, interfaces, and / or code that is operable to identify one or more features from the image data provided to the data transformation processor 140 and the transformed data generated by the data transformation processor 140. The feature recognition processor 150 may identify one or more features only from the transformed data provided by the data transformation processor 140 rather than from the original image data (e.g., spatial B-mode image data). The feature recognition processor 150 may identify features from both the transformed data provided by the data transformation processor 140 and the original image data (e.g., spatial B-mode image data). Such features may be identified by using statistical values (e.g., the likelihood of the presence of a feature in the spatial B-mode image data is 20%, 40%, 60%, or 80%). Multiple features may be identified, and the feature with the greatest likelihood may be selected. For example, if the image data indicates a 95% chance of a malignant lesion and the image data indicates a 5% chance of a benign lesion, then the malignant lesion will be identified.

[0069] The feature recognition processor 150 may implement a machine learning model to identify features in the spatial B-mode image data. Such models may include a convolutional neural network (or CNN). Such convolutional neural network models may be three-dimensional. Figure 7 A machine learning model 700 is illustrated, which is a convolutional neural network. The machine learning model 700 receives a set of M input image data. The input image data may correspond to a limited amount of underlying spatial image data acquired by an ultrasound system. For example, the input image data may correspond to or be limited to a region of interest, such as Figure 2 the region of interest 301 shown.

[0070] To generate a useful feature map, the format of the input image data may be adjusted. As shown, the machine learning model 700 may receive multiple input image data. The image data may include one or more sets of transformed data (e.g., Fourier, shear, and / or Hadamard), and optionally combined with spatial image data (e.g., B-mode, Doppler, color Doppler). One or more different kernels or filters may be applied to the input image data. The kernel may be three-dimensional and may have a depth equal to the depth of the input image data. As Figure 7 shown, each of the input data and the kernel has a depth of three layers, but two-dimensional kernels are also possible. The filtered input image data may produce a corresponding number of feature maps. The feature maps may be two-dimensional (as shown) or one-dimensional. The feature maps may be further processed or evaluated to determine the likelihood of the presence of a given feature (e.g., an organ with a benign or malignant lesion).

[0071] Figure 4An example of training a machine learning model 400 for use by a feature recognition processor 150 is shown. The model 400 can be trained through a supervised learning process in which training data (410, 420, 430, and / or 440) is labeled or annotated. The training can be performed on a training system 200. The training data can be stored in a training database 220 and processed by a training engine 210. Not all of the training data can be stored in the training database 220. Spatial image training data can be stored in the training database 220, and transformed image training data can be determined by a processor such as data transformation processor 140 based on the stored spatial image training data.

[0072] The machine learning model 400 can receive input image data, including spatial B-mode training data 410, Fourier transform training data 420, shear transform training data 430, and / or Hadamard transform training data 440. The machine learning model 400 can be similar to the machine learning model 700. The learning can be performed in two steps. First, the input image data 410, 420, 430, and / or 440 can be input into the machine learning model 400, and then the machine learning model processes the input image data through a neural network to generate an output vector. The highest value of the output vector represents the detected object class, such as a kidney with a malignant lesion. This process can be referred to as feedforward. The detection can be correct or incorrect. Then, a loss function can be determined based on the target value and the actual value of the output of the machine learning model 400. The actual value is known because the training data is labeled or annotated. The loss function can include some or all of the elements and parameters of the neural network. These parameters are updated to minimize or reduce the loss function. This process can be referred to as backpropagation. The feedforward and backpropagation processes can be iterated one or more times until the machine learning model 400 is sufficiently trained. The training can be performed on the training system 200. Then, the trained machine learning model 400 can be stored on the ultrasound system 100.

[0073] Figure 5 FIG. 500 is a flow chart that illustrates exemplary steps that can be used to train a machine learning model to recognize one or more types of features in a spatial B-mode image according to various embodiments. Examples of machine learning models are described in the context of machine learning models 400, 700. The steps in the flow chart 500 can be performed in a different order or some steps can be omitted. The flow chart can be executed by the training engine 210. The training data described in the flow chart 500 can be stored in the training database 220. The machine learning model can be or include a convolutional neural network.

[0074] At step 510, the training of a machine learning model is initiated by a processor such as a processor in the training engine 210. At step 520, spatial B-mode training data (such as data 410) is received at the machine learning model. Exemplary spatial B-mode training data is described in connection with data 310, 410. Instead of or in addition to spatial B-mode training data, other spatial data such as Doppler or color Doppler image data may be used. This data can be labeled or annotated. For example, each image can be labeled as a specific type of organ and whether the organ has an abnormality such as a benign or malignant lesion. Annotation can be performed manually to generate training data for the spatial B-mode training data.

[0075] At step 530, Fourier transform training data (such as data 420) is received at the machine learning model. Exemplary Fourier transform training data is described in connection with data 320, 420. The Fourier transform training data can be labeled or annotated. Annotations or labels from the original spatial image data can be carried over to the Fourier transform training data. The Fourier transform training data can be stored in the training database 220 or can be transformed from the spatial image data stored in the training database 220 before being input into the machine learning model. In other words, it may not be necessary to store the Fourier transform training data in the training database 220.

[0076] At step 540, shear transform training data (such as data 430) is received at the machine learning model. Exemplary shear transform training data is described in connection with data 330, 430. The shear transform training data can be labeled or annotated. Annotations or labels from the original spatial image data can be carried over to the shear transform training data. The shear transform training data can be stored in the training database 220 or can be transformed from the spatial image data stored in the training database 220 before being input into the machine learning model. In other words, it may not be necessary to store the shear transform training data in the training database 220.

[0077] At step 550, Hadamard transform training data (such as data 440) is received at the machine learning model. Exemplary Hadamard transform training data is described in connection with data 340, 440. The Hadamard transform training data can be labeled or annotated. Annotations or labels from the original spatial image data can be carried over to the Hadamard transform training data. The Hadamard transform training data can be stored in the training database 220 or can be transformed from the spatial image data stored in the training database 220 before being input into the machine learning model. In other words, it may not be necessary to store the Hadamard transform training data in the training database 220.

[0078] At step 560, the machine learning model is trained. It can be based on the combinationFigure 4 and 7 the described techniques to train a machine learning model. Training continues until step 570, at which point training is complete. The trained machine learning model can be stored in the ultrasound system 100 and implemented by the feature recognition processor 150.

[0079] Figure 6 is a flowchart 600 that illustrates exemplary steps according to various embodiments that can be used to identify one or more types of features in a spatial B-mode image using a machine learning model (e.g., machine learning models 400, 700). The machine learning model has been trained. The flowchart 600 can be implemented by the ultrasound system 100, which includes a data transformation processor 140 and a feature recognition processor 150.

[0080] At step 610, spatial B-mode image data of a region of interest can be obtained. Techniques for obtaining spatial B-mode image data and determining the region of interest therein were described above in connection with the ultrasound system 100. The above-described exemplary region of interest 310 of the spatial B-mode image data 300 was described in the context of Figure 2 and Figure 3A

[0081] At step 620, the spatial B-mode image data can be transformed into at least one other domain (e.g., Fourier, slant Hadamard). As described above, the transformation can be performed by the data transformation processor 140.

[0082] At step 630, the trained model can be used to identify at least one feature in the spatial B-mode image data. The feature recognition processor 150 can be used to identify the feature. The transformed data of the region of interest and optionally the spatial B-mode image data can be received by the trained machine learning model (such as machine learning models 400, 700). The trained machine learning model can output the likelihood of a given feature—e.g., there is an 80% chance of a malignant lesion appearing on the patient's kidney, a 19% chance of a benign lesion appearing, and a 1% chance of no lesion.

[0083] As used herein, the term "circuitry" refers to physical electronic components (e.g., hardware) and any software and / or firmware ("code") that is configurable hardware, executed by hardware, and / or otherwise associated with the hardware. For example, as used herein, a particular processor and memory can include a first "circuit" when executing one or more first codes, and a second "circuit" when executing one or more second codes. As used herein, "and / or" means any one or more of the items in a list joined by "and / or". For example, "x and / or y" means any element in the three-element set {(x), (y), (x, y)}. As another example, "x, y, and / or z" means any element in the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. As used herein, the term "exemplary" means serving as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "such as" introduce a list of one or more non-limiting examples, instances, or illustrations. As used herein, circuitry "is operable to" and / or "is configured to" perform a function whenever the circuitry includes the necessary hardware and code (if required) to perform the function, regardless of whether the execution of the function is disabled or not enabled by some user-configurable setting.

[0084] Other embodiments may provide a computer-readable device and / or a non-transitory computer-readable medium, and / or a machine-readable device and / or a non-transitory machine-readable medium, on which machine code and / or a computer program having at least one code segment executable by a machine and / or a computer are stored, such that the machine and / or the computer perform the steps described herein for enhancing sequential ultrasound images using deep learning.

[0085] Accordingly, the present disclosure may be implemented in hardware, software, or a combination of hardware and software. The present disclosure may be implemented in a centralized manner in at least one computer system or in a distributed manner in which different elements are distributed among several interconnected computer systems. Any kind of computer system or other device suitable for performing the methods described herein is appropriate.

[0086] Each embodiment may also be embedded in a computer program product that includes all the features capable of implementing the methods described herein and, when loaded into a computer system, is capable of executing these methods. A computer program herein refers to any expression of a set of instructions represented in any language, code, or notation, which are intended to cause a system with information processing capabilities to directly perform a specific function or to perform a specific function after one or both of the following: a) being converted into another language, code, or notation; b) being reproduced in a different physical form.

[0087] Although the present disclosure has been described with reference to certain embodiments, those skilled in the art should understand that various changes can be made and equivalents can be substituted without departing from the scope of the present disclosure. Additionally, many modifications can be made to adapt a particular situation or material to the teachings of the present disclosure without departing from its scope. Therefore, the present disclosure is not intended to be limited to the particular embodiments disclosed, but the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A method for analyzing ultrasonic image data (300) obtained from ultrasonic imaging, the method comprising: obtaining the ultrasonic image data by an ultrasonic probe (104); transforming the ultrasonic image data by a processor (140) using at least one transform to generate at least one set (320, 330, 340) of transformed data; inputting the ultrasonic image data and at least one set of the transformed data into a machine learning model (400), wherein the machine learning model is implemented by the processor (150); implementing the machine learning model by the processor using the ultrasonic image data and at least one set of the transformed data; and identifying, by the processor, at least one feature in the ultrasonic image data as determined by the machine learning model.

2. The method according to claim 1, the method further comprising: Determining a range of the ultrasonic image data according to a region of interest (301).

3. The method according to claim 1, wherein the machine learning model comprises a convolutional neural network.

4. The method according to claim 3, wherein the method further comprises: The machine learning model is trained by: inputting annotated ultrasonic image data (410, 420, 430, 440) into the machine learning model, wherein the ultrasonic image data indicates the presence or absence of the at least one feature; transforming the ultrasonic image data using at least one transform to generate transformed data; and inputting the transformed data into the machine learning model.

5. The method according to claim 4, the method further comprising: When the machine learning model receives additional ultrasonic image data indicating the presence or absence of the at least one feature, updating the machine learning model to reduce a loss function.

6. The method according to claim 1, wherein the ultrasonic image data comprises spatial B-mode image data.

7. The method according to claim 1, wherein at least one set of the transformed data comprises at least one of Fourier transform data (320), shear transform data (330), or Hadamard transform data (340).

8. The method according to claim 1, wherein the at least one feature comprises at least one of a benign lesion or a malignant lesion in an organ of a patient.

9. A system (100) for analyzing ultrasonic image data (300) obtained from ultrasonic imaging, the system comprising: an ultrasonic probe (104) configured to obtain the ultrasonic image data; and a processor (132) configured to: transform the ultrasonic image data using at least one transform to generate at least one set (320, 330, 340) of transformed data; input the ultrasonic image data and at least one set of the transformed data into a machine learning model (400), wherein the machine learning model is implemented by the processor (132); implement the machine learning model using the ultrasonic image data and at least one set of the transformed data; and identify at least one feature in the ultrasonic image data as determined by the machine learning model.

10. The system according to claim 9, wherein the machine learning model comprises a convolutional neural network.

11. The system according to claim 10, wherein the processor is further configured to train the machine learning model by: inputting annotated ultrasound image data (410, 420, 430, 440) into the machine learning model, wherein the ultrasound image data indicates the presence or absence of the at least one feature; transforming the ultrasound image data using at least one transformation to generate transformed data; and inputting the transformed data into the machine learning model.

12. The system according to claim 11, wherein the processor is further configured to implement the machine learning model to reduce a loss function when the machine learning model receives additional ultrasound image data indicating the presence or absence of the at least one feature.

13. The system according to claim 9, wherein the ultrasound image data comprises spatial B-mode image data.

14. The system according to claim 9, wherein at least one set of the transformed data comprises at least one of Fourier transform data (320), shear transform data (330), or Hadamard transform data (340).

15. The system according to claim 9, wherein the at least one feature comprises at least one of a patient's organ having a benign lesion or a malignant lesion.

Images