Techniques for generating enhanced sequential image data

Abstract

The disclosure relates to techniques for generating enhanced sequential image data. A method for enhancing sequential ultrasound image data in an ultrasound system (100), the method comprising: acquiring, by an ultrasound probe (104) of the ultrasound system (100), first image data during a first time period (230) and third image data during a third time period (240) after the first time period; and inferring inferred image data corresponding to a second time period, where the second time period is between the first time period and a third time period, where the inferred image data (360, 460) is inferred by: inputting the degraded image data (340, 440) into a trained machine learning model (210); and generating inferred image data (360, 460) through the trained machine learning model (210).

Description

Technical Field

[0001] Certain embodiments relate to ultrasound imaging. More specifically, certain embodiments relate to techniques for enhancing Doppler imaging of time-series processes, such as blood flow imaging. Background Art

[0002] Ultrasound imaging is a medical imaging technique used to image human anatomy. Ultrasound imaging can be used to image or analyze blood flow through a patient's cardiovascular system. Ultrasound imaging uses real-time, non-invasive, high-frequency sound waves to produce two-dimensional (2D), three-dimensional (3D), and / or four-dimensional (4D) (i.e., real-time / continuous 3D image data) image data.

[0003] Some ultrasound imaging systems may have the capability to perform imaging using a variety of modes such as B-mode, Doppler mode, and / or color Doppler mode. According to some techniques, the ultrasound imaging system may implement different techniques sequentially over a period of time. For example, the ultrasound imaging system may obtain B-mode image data at a first time, Doppler image data at a subsequent second time, B-mode image data at a subsequent third time, Doppler image data at a subsequent fourth time, and so on. In this example, Doppler image data is not obtained at the first time and the third time. If the patient's blood flow is imaged, data loss will occur because the patient's blood is constantly moving and the distribution is different at each of the four times. Each of these "times" may refer to a substantially instantaneous time period and / or an extended time period. A given extended time period may include multiple time slots or frames, which include corresponding multiple sequence image data. The duration of these "times" may be equal or different.

[0004] Further limitations and disadvantages of conventional and traditional approaches will become apparent to those skilled in the art by comparing such systems with certain aspects of the present disclosure as set forth in the remainder of this application with reference to the accompanying figures. Summary of the invention

[0005] According to an embodiment, a method for enhancing sequential ultrasound image data in an ultrasound system includes: acquiring first image data during a first time period and acquiring third image data during a third time period after the first time period by an ultrasound probe of the ultrasound system; and inferring inferred image data corresponding to a second time period, wherein the second time period is between the first time period and the third time period, wherein the inferred image data is inferred by: inputting degraded image data into a trained machine learning model; and generating the inferred image data by the trained machine learning model. The first image data, the third image data, and the inferred image data may include Doppler image data. The degraded image data may be obtained by mixing noise with the first image data. The inferred image data may include time domain data. The inferred Doppler image data may include frequency domain data. The trained machine learning training model may include a generative adversarial network (GAN) model. The GAN model may include a generator and a discriminator, and may be trained by: inputting degraded image data into the generator; outputting, by the generator, samples corresponding to the degraded image data to the discriminator; inputting reference image data into the discriminator; and discriminating, using the discriminator, the degraded image data and the reference image data according to at least one function. The samples may include a probability density function of a real sample of the degraded image data. The step of discriminating using the discriminator may include minimizing a loss function. The reference image data may be obtained from imaging of a phantom device. The empirical training Doppler image data may be obtained from Doppler imaging of a human. The degraded image data may be generated by mixing noise with the first image data. The method may also include acquiring B-mode data during a second time period.

[0006] According to an embodiment, a system for enhancing sequential ultrasound image data includes: an ultrasound probe configured to obtain first image data during a first time period and third image data during a third time period after the first time period; and a processor configured to infer inferred image data corresponding to a second time period, wherein the second time period is between the first time period and the third time period, wherein the inferred image data is inferred by: inputting degraded image data into a trained machine learning model; and generating the inferred image data by the trained machine learning model. The first image data, the third image data, and the inferred image data may include Doppler image data. The processor may also be configured to generate degraded image data by mixing noise with the first image data. The trained machine learning training model may include a generative adversarial network (GAN) model. The processor may also be configured to implement a GAN model using a generator and a discriminator, and wherein the processor is further configured to train the GAN model by: inputting degraded image data into the generator; outputting samples corresponding to the degraded image data to the discriminator through the generator; inputting reference image data into the discriminator; and using the discriminator to discriminate the degraded image data and the reference image data according to at least one function. The degraded image data may be generated by mixing noise with the first image data. The ultrasound probe may also be configured to acquire B-mode data during a second time period.

[0007] These and other advantages, aspects and novel features of the present disclosure, as well as details of illustrated embodiments thereof, will be more fully understood from the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram of an exemplary ultrasound system operable to enhance sequential ultrasound image data using a machine learning model, according to various embodiments.

[0009] Figure 2 It is a diagrammatic representation of an incomplete series of image data in which image data at a specific time is not acquired.

[0010] Figure 3 is an exemplary flowchart for training a machine learning model to generate enhanced image sequence data in the time domain according to various embodiments.

[0011] Figure 4 is an exemplary flowchart for training a machine learning model to generate enhanced image sequence data in the frequency domain according to various embodiments.

[0012] Figure 5is a flow chart illustrating exemplary steps that may be used to train a machine learning model for enhancing sequential ultrasound image data, according to various embodiments.

[0013] Figure 6 is a flow chart illustrating exemplary steps that may be used to enhance ultrasound image sequence data using a machine learning model, according to various embodiments. DETAILED DESCRIPTION

[0014] Certain embodiments may exist in a method and system for enhancing time series ultrasound imaging such as blood flow imaging using machine learning. Aspects of the present disclosure have the following technical effects, namely, using machine learning to enhance time series imaging to help provide a diagnosis. Various embodiments have the following technical effects, namely, using machine learning to process the acquired ultrasound image data to enhance time series ultrasound image data. Certain embodiments have the following technical effects, namely, predicting image data for a given time period when ultrasound image data was not obtained at that time. Certain embodiments have the following technical effects, namely, predicting ultrasound image data (e.g., Doppler image data) for a given time period when ultrasound image data was not obtained at that time. Aspects of the present disclosure have the following technical effects, namely, generating improved ultrasound time series image data in an ultrasound imaging system that sequentially uses multiple imaging modes such as B mode, Doppler mode, and color Doppler mode over time. Aspects of the present disclosure have the following technical effects, namely, using degraded image data (e.g., degraded Doppler image data) to predict the blood flow of a patient during a period when image data (e.g., Doppler image data) is not obtained.

[0015] When reading in conjunction with the accompanying drawings, the following specific embodiments of the aforementioned invention content and certain embodiments will be better understood. In terms of the scope of the figures of the functional blocks of each embodiment shown in the accompanying drawings, these functional blocks do not necessarily represent the division between hardware circuits. Therefore, for example, one or more functional blocks (e.g., processors or memories) can be implemented in a single piece of hardware (e.g., a general signal processor or random access memory block, a hard disk, etc.) or multiple pieces of hardware. Similarly, a program can be a stand-alone program, can be incorporated into an operating system as a subroutine, can be a function in an installed software package, etc. It should be understood that each embodiment is not limited to the arrangement and tools shown in the accompanying drawings. It should also be understood that embodiments can be combined, or other embodiments can be utilized, and structural, logical and electrical changes can be made without departing from the scope of various embodiments. Therefore, the following detailed description should not be regarded as a restrictive meaning, and the scope of the present disclosure is limited by the attached claims and their equivalents.

[0016] As used herein, elements or steps listed in the singular and beginning with the word "one" or "an" should be understood as not excluding a plurality of said elements or steps, unless such exclusion is explicitly stated. In addition, references to "exemplary embodiments," "various embodiments," "certain embodiments," "representative embodiments," etc. are not intended to be interpreted as excluding additional embodiments that also include the features of the narrative. In addition, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" an element or multiple elements with a particular attribute may include additional elements that do not have that attribute.

[0017] In addition, as used herein, the term "image" refers broadly to both visible images and data (image data) characterizing visible images. However, many embodiments generate (or are configured to generate) at least one visible image. In addition, as used herein, the phrase "image" is used to refer to an ultrasound mode, which can be one-dimensional (1D), two-dimensional (2D), three-dimensional (3D) or four-dimensional (4D), and includes brightness mode (B mode), motion mode (M mode), color motion mode (CM mode), color flow mode (CF mode), pulsed wave (PW) Doppler, continuous wave (CW) Doppler, contrast enhanced ultrasound (CEUS) and / or B mode and / or CF mode sub-mode, such as harmonic imaging, shear wave elastic imaging (SWEI), strain elastic imaging, tissue velocity imaging (TVI), power Doppler imaging (PDI), B flow, microangiography (MVI), ultrasound-guided attenuation parameters (UGAP), etc.

[0018] Furthermore, as used herein, the term processor or processing unit refers to any type of processing unit that can perform the required computations required by various embodiments, such as a single-core or multi-core CPU, an accelerated processing unit (APU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination thereof.

[0019] It should be noted that various embodiments of generating or forming an image described herein may include a process for forming an image that includes beamforming in some embodiments and does not include beamforming in other embodiments. For example, an image may be formed without beamforming, such as by multiplying a matrix of demodulated data by a matrix of coefficients such that the product is an image and wherein the process does not form any "beams." Additionally, the formation of an image may be performed using a combination of channels that may originate from more than one transmit event (e.g., synthetic aperture techniques).

[0020] In various embodiments, ultrasound processing to form images, including ultrasound beamforming, such as receive beamforming, is performed, for example, in software, firmware, hardware, or a combination thereof. One specific implementation of an ultrasound system having a software beamformer architecture formed according to various embodiments is described in Figure 1 Shown in.

[0021] Figure 1 is a block diagram of an exemplary ultrasound system operable to automatically place a medical device in an anatomical structure using a locking mechanism, according to various embodiments. Figure 1 , an ultrasound system 100 and a training system 200 are shown. The ultrasound system 100 includes a transmitter 102, an ultrasound probe 104, a transmit beamformer 110, a receiver 118, a receive beamformer 120, an analog-to-digital (A / D) converter 122, a radio frequency (RF) processor 124, an RF quadrature (RF / IQ) buffer 126, a user input device 130, a signal processor 132, an image buffer 136, a display system 134, and an archive 138.

[0022] The transmitter 102 may include suitable logic components, circuits, interfaces and / or codes that are operable to drive the ultrasound probe 104. The ultrasound probe 104 may be a linear, convex, intracavitary or phased array transducer. The ultrasound probe 104 may include a two-dimensional (2D) array of piezoelectric elements. The ultrasound probe 104 may include a group of transmitting transducer elements 106 and a group of receiving transducer elements 108 that generally constitute the same element. The group of transmitting transducer elements 106 may emit an ultrasound signal that passes through the oil and the probe cap and enters the target. In a representative embodiment, the ultrasound probe 104 is operable to collect ultrasound image data covering at least a substantial portion of an anatomical structure, such as a heart, an ovary or any suitable anatomical structure. In an exemplary embodiment, the ultrasound probe 104 may be operated in a volume acquisition mode, wherein the transducer assembly of the ultrasound probe 104 acquires a plurality of parallel 2D ultrasound slices that form an ultrasound volume.

[0023] The transmit beamformer 110 may include suitable logic, circuitry, interfaces, and / or code operable to control the transmitter 102 to drive the set of transmit transducer elements 106 via the transmit sub-aperture beamformer 114 to transmit ultrasound transmit signals into a region of interest (e.g., a person, an animal, an underground cavity, a physical structure, etc.). The transmitted ultrasound signals may be backscattered from structures in the object of interest (such as blood cells or tissue) to generate echoes. The echoes are received by the receive transducer elements 108.

[0024] The set of receive transducer elements 108 in the ultrasound probe 104 are operable to convert received echoes into analog signals, sub-aperture beamformed by the receive sub-aperture beamformer 116, and then transmitted to the receiver 118. The receiver 118 may include suitable logic, circuitry, interfaces, and / or code that may be operable to receive the signals from the receive sub-aperture beamformer 116. The analog signals may be transmitted to one or more of the plurality of A / D converters 122.

[0025] The plurality of A / D converters 122 may include suitable logic components, circuits and interfaces and / or codes that are operable to convert analog signals from the receiver 118 into corresponding digital signals. The plurality of A / D converters 122 are disposed between the receiver 118 and the RF processor 124. Nevertheless, the present disclosure is not limited in this regard. Therefore, in some embodiments, the plurality of A / D converters 122 may be integrated within the receiver 118.

[0026] The RF processor 124 may include suitable logic components, circuits, interfaces, and / or codes that may be operable to demodulate the digital signals output by the plurality of A / D converters 122. According to an embodiment, the RF processor 124 may include a complex demodulator (not shown) that may be operable to demodulate the digital signals to form I / Q data pairs representing corresponding echo signals. The RF or I / Q signal data may then be transferred to the RF / IQ buffer 126. The RF / IQ buffer 126 may include suitable logic components, circuits, interfaces, and / or codes that may be operable to provide temporary storage of the RF or I / Q signal data generated by the RF processor 124.

[0027] The receive beamformer 120 may comprise suitable logic, circuitry, interfaces, and / or code operable to perform digital beamforming processing, such as summing delayed channel signals received from the RF processor 124 via the RF / IQ buffer 126 and outputting a beam summed signal. The resulting processed information may be a beam summed signal output from the receive beamformer 120 and communicated to the signal processor 132. According to some embodiments, the receiver 118, the plurality of A / D converters 122, the RF processor 124, and the beamformer 120 may be integrated into a single beamformer, which may be a digital beamformer. In various embodiments, the ultrasound system 100 includes a plurality of receive beamformers 120.

[0028] The user input device 130 may be used to enter patient data; scan parameters; settings; select a protocol and / or template; select a target structure for image acquisition; enter and / or select a region of interest; modify a region of interest; select a region of interest for image acquisition, focus / zoom volume; etc. In an exemplary embodiment, the user input device 130 may be operable to configure, manage, and / or control the operation of one or more components and / or modules in the ultrasound system 100. In this regard, the user input device 130 may be operable to configure, manage, and / or control the operation of the transmitter 102, ultrasound probe 104, transmit beamformer 110, receiver 118, receive beamformer 120, RF processor 124, RF / IQ buffer 126, user input device 130, signal processor 132, image buffer 136, display system 134, and / or archive 138. The user input device 130 may include buttons, rotary encoders, touch screens, motion tracking, voice recognition, a mouse device, a keyboard, a camera, and / or any other device capable of receiving user instructions. In certain embodiments, one or more of the user input devices 130 may be integrated into other components, such as the display system 134 or the ultrasound probe 104, for example. For example, the user input device 130 may include a touch screen display.

[0029] The signal processor 132 may include suitable logic components, circuits, interfaces and / or codes that are operable to process ultrasound scan data (e.g., summed IQ signals) to generate ultrasound images for presentation on the display system 134. The signal processor 132 is operable to perform one or more processing operations based on multiple ultrasound modalities (such as B-mode, Doppler modality, and color Doppler modality) of the acquired ultrasound scan data. In an exemplary embodiment, the signal processor 132 may be used to perform display processing and / or control processing, etc. When the echo signal is received, the acquired ultrasound scan data, such as a series of image data corresponding to blood flow in the region where the patient's anatomical structure is located, may be processed in real time during the scanning session. Additionally or alternatively, the ultrasound scan data may be temporarily stored in the RF / IQ buffer 126 during the scanning session and processed in a less real-time manner in an online operation or an offline operation. In various embodiments, the processed image data may be presented at the display system 134 and / or may be stored at the archive 138. Archive 138 may be a local archive, a Picture Archiving and Communication System (PACS), or any suitable device for storing images and related information.

[0030] The signal processor 132 may be one or more central processing units, microprocessors, microcontrollers, etc. For example, the signal processor 132 may be an integrated component, or may be distributed in various locations. In an exemplary embodiment, the signal processor 132 may include a sequence image acquisition processor 140, a sequence image generator processor 150, and a sequence image discriminator processor 160. The signal processor 132 may be capable of receiving input information from the user input device 130 and / or the archive 138, generating output that may be displayed by the display system 134, and manipulating the output in response to the input information from the user input device 130, etc. For example, the signal processor 132, the sequence image acquisition processor 140, and the sequence image generator processor 150 may be capable of executing any of the methods and / or instruction sets discussed herein according to various embodiments.

[0031] The ultrasound system 100 may be operable to continuously acquire ultrasound scan data at a frame rate suitable for the imaging situation under consideration. Typical frame rates are in the range of 20 to 120 per second, but may be lower or higher. As used herein, "time" or "time period" may correspond to one or more frames. The acquired ultrasound scan data may be displayed on the display system 134 at a display rate that is the same as the frame rate, or slower or faster than the frame rate. A series of images (e.g., of a patient's blood flow) may be displayed simultaneously. An image buffer 136 is included to store processed frames of acquired ultrasound scan data that are not scheduled for immediate display. Preferably, the image buffer 136 has sufficient capacity to store ultrasound scan data frames equivalent to at least several minutes. The frames of ultrasound scan data are stored in a manner that is easy to retrieve from them according to their acquisition order or time. The image buffer 136 may be embodied as any known data storage medium.

[0032] The signal processor 132 may include a sequence image acquisition processor 140 including suitable logic components, circuit systems, interfaces and / or code operable to obtain sequence ultrasound image data using the ultrasound probe 104. In an exemplary embodiment, the sequence image acquisition processor 140 may be configured to capture a series of ultrasound images at target locations of the patient's anatomical structure, such as blood flow in a particular region of interest of the patient's cardiovascular system. For example, the sequence image acquisition processor 140 may be configured to receive user input selecting a region of interest before performing ultrasound image acquisition and analyzing ultrasound image data and / or volumes acquired by the ultrasound images to obtain a sequence of images over time, such as a series of images of a patient's blood flow.

[0033] Figure 2is an exemplary incomplete series of ultrasound image data. The image data may be Doppler image data, but may also be other types of image data. The image data may be image data of blood flow in a region of interest of a patient, and may be obtained by the ultrasound system 100. As shown, two image data sets 230, 240 are obtained at two different times. However, image data for a specific time period between the image data sets 230, 240 is not obtained. For example, when the ultrasound system 100 switches between various modes over time, there may be no image data. According to one technique, the ultrasound system 100 uses multiple modes to image a patient over time. One such example is that the ultrasound system 100 obtains Doppler images and B-mode images in an alternating order of Doppler, B-mode, Doppler, B-mode, etc. During the time when the ultrasound system 100 is obtaining B-mode images, the ultrasound system 100 may not be obtaining Doppler images. Therefore, in conditions such as Figure 2 There may be gaps in a series of Doppler image data such as the one shown. For another example, when the ultrasound system is acquiring color Doppler image data, there may be gaps in the Doppler image data. For another example, image data at a particular time may be ignored or removed for any given reason. In such cases, there may also be gaps.

[0034] Reference again Figure 1 , the sequence image acquisition processor 140 may be configured to collect ultrasound image data as the ultrasound probe 104 slides over a region of interest, anatomical structure, tissue, and / or fluid contained therein (such as blood flowing through a region of interest of a patient's cardiovascular system). As the ultrasound probe 104 slides over such a region, the sequence image acquisition processor 140 collects ultrasound images and arranges them in a chronological order (e.g., from left to right), wherein the first image acquired is the first in the sequence, the second image acquired is the second in the sequence, and so on. For example, the sequence image acquisition processor 140 may generate Figure 2 As mentioned, as the ultrasound probe 104 slides over the region of interest, there may be gaps in the sequence of ultrasound images for a duration when no image data is acquired or when undesirable image data is acquired.

[0035] The sequence ultrasound image data with gaps may be provided by the sequence image acquisition processor 140 to the sequence image generator processor 150. Additionally and / or alternatively, the generated images may be stored at the archive 138 and / or any suitable computer-readable medium, and the sequence image generator processor 150 may obtain the sequence ultrasound image data from the archive 138 and / or any suitable computer-readable medium. In some examples, the sequence image acquisition processor 140 may also be used to generate reference sequence ultrasound image data to be stored in the archive 138, the training database 220, and / or any suitable computer-readable medium.

[0036] The sequence image generator processor 150 may include suitable logic, circuitry, interfaces, and / or code that may be operable to obtain acquired sequence ultrasound image data from the sequence image acquisition processor 140 and / or from the archive 138, the training database 220, and / or any suitable computer readable medium. For example, the sequence image generator processor 150 may be configured to receive acquired sequence ultrasound images with gaps from the sequence image acquisition processor 140, or to retrieve acquired sequence ultrasound images with gaps from the archive 138 and / or the training database 220 and / or any suitable data storage medium.

[0037] The sequence image generator processor 150 may receive sequence image data in which gaps have been filled with degraded image data (hereinafter referred to as partially degraded sequence image data). The degraded image data may be generated by mixing noise with image data obtained before (e.g., immediately before) the gap period. The degraded image data may then be inserted into the gap. The partially degraded sequence image data may be obtained by the sequence image generator processor 150 from the archive 138, the training database 220, and / or any suitable data storage medium.

[0038] refer to Figure 1, the sequence image generator processor 150 may include suitable logic components, circuit systems, interfaces and / or codes, which may be configured to obtain partially degraded sequence ultrasound image data as input and learn probability distributions to generate sample sequence ultrasound image data. For example, the sequence image generator processor 150 may obtain the partially degraded sequence image data from the sequence image acquisition processor 140, the archive 138 and / or any other suitable computer-readable medium. The sequence image generator processor 150 generates sample sequence ultrasound image data that may appear to be real from the partially degraded sequence image data, which may then be provided to the sequence image discriminator processor 160. Additionally or alternatively, the sample sequence image data may be stored in the archive 138, the training database 220 and / or any other suitable computer-readable medium, and the sequence image discriminator processor 160 may retrieve the sample sequence ultrasound images in the archive 138, the training database 220 and / or any other suitable computer-readable medium.

[0039] Still reference Figure 1 , the sequence image discriminator processor 160 may obtain as input reference sequence ultrasound image data from the sequence image generator processor 150, the training database 220, the archive 138, and / or any other suitable computer-readable medium. The sequence image discriminator processor 160 may also be configured to obtain sample sequence ultrasound images from the sequence image generator processor 150, and use the sample sequence ultrasound images to learn to distinguish between reference images and sample images. The sequence image generator processor 150 may also be operable to obtain reference sequence ultrasound image data from the training database 220, from the archive 138, and / or from any suitable data storage medium.

[0040] The reference sequence ultrasound image data may be sequence image data without gaps. The reference sequence ultrasound image data without gaps may be collected in advance. The reference sequence ultrasound image data may be collected in advance from a patient or from a phantom device simulating a fluid flow. The reference sequence ultrasound image data may be collected under a variety of situations and / or conditions. For example, reference sequence ultrasound image data may be collected under different regions of interest, different blood pressures, different blood velocities, different inner diameters corresponding to different regions of interest, different blood components, different viscosities, and the like. Some or all of such reference sequence ultrasound image data may be used for training. The reference sequence ultrasound image data may be stored in the training database 220, the archive 138, and / or any suitable data storage medium.

[0041] In some examples, the machine learning model or technique is a generative adversarial network (GAN) model, and the GAN model is used to train the sequence image generator processor 150 and the sequence image discriminator processor 160. Figure 3 , the generator 310 may be implemented at least in part by the sequence image generator processor 150. In addition, the discriminator 320 may be implemented at least in part by the sequence image discriminator processor 160. The generator 310 may be configured to generate sample sequence ultrasound image data by mapping the partially degraded ultrasound image data to a latent space to learn a probability distribution, which the generator 310 may use to generate sample sequence ultrasound images that may appear to be real. At least this is the object of the generator 310, i.e., to generate sample sequence ultrasound image data that appears to be real from partially degraded sequence ultrasound image data.

[0042] The goal of generator 310 may be to learn the distribution p θ (x), which is similar to the distribution of the reference sequence ultrasound image data p r (x). The generator 310 may generate a G The probability density function of the sample sequence ultrasound image data (x) is substantially equal to the probability density function p of the reference sequence ultrasound image data. r (x). Differentiable function p θ (x) can be learned by the generator 310 so that p θ (x)>0 and and optimized by maximum likelihood. Additionally or alternatively, as a non-limiting example, the generator 310 may learn p θ The differential transformation function q of (x) θ (z), and is optimized by maximum likelihood that z is an existing common distribution such as a uniform distribution or a Gaussian distribution.

[0043] The discriminator 320 may receive as input the sample sequence ultrasound image data from the generator 310, and may also receive as input the reference sequence ultrasound image data. The discriminator 320 may learn to distinguish between these two inputs. The discriminator 320 may classify the image as a real image or a generated image (from the generator 310) by outputting a value D(x) 330. In some examples, this value is a binary value (e.g., "0" or "1"). The discriminator 320 may identify the image from the real data distribution p r (x), where function D indicates the data point x i ∈R n In the case of binary classification, if the estimated probability D(x i ):->R n [0,1] is the positive class p i And 1-D(x i ):->R[0,1] is the negative class q i , then p i With q i The cross entropy distribution between For a given point x i and the corresponding label y i , data distribution x i Can come from real data x i ~p r (x) or generator data x i ~p g (z). The generator 310 and the discriminator 320 may have an adversarial relationship, wherein the generator 310 produces fake generated image data and the discriminator 320 learns to distinguish between real image data and generated image data. Given that half of the data from the generator 310 and the discriminator are real data and generated data, the generator 310 and the discriminator 320 may compete with each other in a min-max game to minimize the loss function.

[0044] D(x) 330 may be provided as feedback to the discriminator 320 and the generator 310. Additionally and / or alternatively, feedback may be provided to the generator 310 and the discriminator 320 using one or more cost functions and / or loss functions.

[0045] For example, the loss function can be as follows:

[0046]

[0047] in To solve or overcome the vanishing gradient effect.

[0048] By competing and receiving feedback, the generator 310 can generate images similar to real ultrasound images, and the discriminator becomes better at distinguishing between real ultrasound images and generated ultrasound images. The goal is to make the images generated by the generator 310 generate ultrasound images similar to real ultrasound images, at which point the discriminator 320 may also not be able to distinguish between real ultrasound images and generated ultrasound images, and the training can be considered complete.

[0049] Once the training of the generator 310 and the discriminator 320 using the GAN model is completed, the generator 310 may generate enhanced image data, such as enhanced Doppler image data. The image data filling the gap may be inferred image data (e.g., inferred Doppler image data), and may be inferred by the trained generator 310 after the generator 310 receives degraded image data (e.g., image data included in partially degraded sequence image data, such as degraded Doppler image data).

[0050] Figure 3 Input image data in the time domain is shown. The input image data can be represented as an I component and a Q component. Figure 4 Can be similar to Figure 3 , except that the processed image data may be in the frequency domain. Generator 410 may be similar to generator 310, except that generator 410 performs processing on frequency domain data, while generator 310 performs processing on time domain data. Discriminator 420 may be similar to discriminator 320, except that discriminator 420 performs processing on frequency domain data, while discriminator 320 performs processing on time domain data. Element 430 may be similar to element 330, except that element 430 relates to frequency domain data, while element 330 relates to time domain data. Partially degraded sequence image data 440 may be similar to partially degraded sequence image data 340, except that the former is in the frequency domain, while the latter is in the time domain. Reference sequence image data 450 may be similar to reference sequence image data 350, except that the former is in the frequency domain, while the latter is in the time domain. Enhanced sequence image data 460 may be similar to enhanced sequence image data 360, except that the former is in the frequency domain, while the latter is in the time domain.

[0051] Refer again Figure 1 The display system 134 may be any device capable of conveying visual information to a user. For example, the display system 134 may include a liquid crystal display, a light emitting diode display, and / or any suitable one or more displays. The display system 134 may be operable to present a 2D ultrasound image, a 2D sequential ultrasound image, a bi-plane ultrasound image, a bi-plane ultrasound slice extracted from a 3D / 4D volume, a rendered 3D / 4D volume, a selectable target structure, and / or any suitable information.

[0052] The archive 138 may be one or more computer-readable memories, such as a picture archiving and communication system (PACS), a server, a hard disk, a floppy disk, a CD, a CD-ROM, a DVD, a compact memory, a flash memory, a random access memory, a read-only memory, an electrically erasable and programmable read-only memory, and / or any suitable memory, that are integrated with the ultrasound system 100 and / or communicatively coupled (e.g., via a network) to the ultrasound system 100. The archive 138 may include, for example, a database, a library, a collection of information, or other memory that is accessed by and / or associated with the signal processor 132. For example, the archive 138 may be capable of storing data temporarily or permanently. The archive 138 may be capable of storing medical image data, data generated by the signal processor 132, and / or instructions readable by the signal processor 132, etc. In various embodiments, for example, the archive 138 stores 2D ultrasound images, 2D sequential ultrasound images, bi-plane ultrasound images, bi-plane ultrasound slices extracted from a 3D / 4D volume, rendered 3D / 4D volumes, instructions for acquiring ultrasound image data, instructions for generating sequential ultrasound images, instructions for generating sample sequential ultrasound images, instructions for classifying images as generated images or real images, instructions for providing feedback based on the classification of images, instructions for determining that a target function has been achieved, and instructions for generating enhanced sequential ultrasound images.

[0053] The components of the ultrasound system 100 may be implemented in software, hardware, firmware, etc. The various components of the ultrasound system 100 may be communicatively connected. The components of the ultrasound system 100 may be implemented separately and / or integrated in various forms. For example, the display system 134 and the user input device 130 may be integrated into a touch screen display.

[0054] Still refer to Figure 1, the training system 200 may include a training engine 210 and a training database 220. The training engine 210 may include suitable logic components, circuit systems, interfaces and / or codes that are operable to train neurons of a deep neural network (e.g., an artificial intelligence model) inferred (i.e., deployed) by the sequence image acquisition processor 140, the sequence image generator processor 150, and / or the sequence image generator processor 160. For example, the artificial intelligence model inferred by the sequence image generator processor 150 and / or the sequence image discriminator processor 160 may be trained to automatically acquire ultrasound images and / or volumes using a database 220 of classified ultrasound images (e.g., including sequence image data) and / or volumes of anatomical structures. For another example, the artificial intelligence model inferred by the sequence image acquisition processor 140 may be trained to automatically identify a target structure, surrounding structures, target structure shape, major axis / minor axis of a target structure, etc. depicted in an ultrasound volume using a database 220 of classified ultrasound volumes of possible target structures. For another example, the classification may correspond to one or more aspects or parameters of blood flow, as described above.

[0055] In various embodiments, the database 220 of training images may be a picture archiving and communication system (PACS) or any suitable data storage medium. In certain embodiments, the training engine 210 and / or the training image database 220 may be a remote system communicatively coupled to the ultrasound system 100 via a wired or wireless connection, such as Figure 1 Additionally and / or alternatively, components or all of the training system 200 may be integrated with the ultrasound system 100 in various forms. In some examples, the training image database may include reference sequence ultrasound images of anatomical structures and / or tissues. In some examples, the reference sequence ultrasound images may be generated by the sequence image acquisition processor 140 and provided to the training image database 220.

[0056] Figure 5 5 is a flowchart 500 showing exemplary steps 510 to 580 that may be used to train a machine learning model for enhancing sequential ultrasound images according to various embodiments. Certain embodiments may omit one or more steps, and / or perform the steps in a different order than listed, and / or combine certain steps discussed below. For example, some steps may not be performed in certain embodiments. For another example, certain steps may be performed in a different time order than listed below, including simultaneously.

[0057] At step 510, the signal processor 132 of the ultrasound system 100 may be configured to initiate training of enhanced sequence ultrasound images. For example, at step 520, the sequence image identification processor 160 may be configured to receive reference sequence ultrasound image data, for example, from the sequence image acquisition processor 140, the archive 138, the training database 220, and / or other suitable memory. The reference sequence ultrasound image data may be generated according to the techniques described herein.

[0058] At step 530, the sequence image generator processor 150 may receive degraded sequence image data, such as partially degraded sequence image data. The partially degraded sequence image data may include real components at certain times and degraded image data at times between the real components. The partially degraded sequence image data may include alternating real image data sets with degraded image data sets. The partially degraded sequence image data may have only one real image data set at a first time and a degraded image data set at a second time. The real image data may be provided by the sequence image acquisition processor 140, the archive 138, the training database 220 and / or other suitable memory. The degraded image data may be image data mixed with noise (e.g., random noise). The image data used to generate the degraded image data may be obtained from a time period before a gap (e.g., immediately before a gap). An example of such image data is Figure 2 The duration of the image data may be substantially the same as the duration of the gap.

[0059] At step 540, the signal processor 132 may be configured to generate sample sequence ultrasound images using the degraded (and / or partially degraded) sequence ultrasound image data. For example, the sequence ultrasound image generator processor 150 may be configured to map the sample sequence ultrasound images to a latent space and learn a probability distribution that the sequence image generator processor 150 may use to generate the sample sequence ultrasound images. The sequence image generator processor 150 may provide the sample sequence ultrasound images to the sequence image discriminator processor 160.

[0060] At step 550, the signal processor 132 may be configured to discriminate or classify the sample sequence ultrasound images as real images or generated images. For example, the sequence image discriminator processor 160 may receive the sample sequence ultrasound images from the sequence image generator processor 150, and the reference sequence ultrasound images as input. The sequence image discriminator processor 160 may classify the received images as real images or generated images.

[0061] At step 560, the signal processor 132 may be configured to use the results of the classification in order to provide feedback to the sequence image generator processor 150 and the sequence image discriminator processor 160. For example, the sequence image discriminator processor 160 may classify the received image as a real image or a generated image by outputting a value D(x). In some examples, the value is a binary value (e.g., "0" or "1"). D(x) may be provided as feedback to the sequence image discriminator processor 160 and the sequence image generator processor 150. Additionally and / or alternatively, one or more cost functions and / or loss functions may be used to provide feedback to the sequence image generator processor 150 and the sequence image discriminator processor 160.

[0062] At step 570, the signal processor 132 may be configured to repeat steps 540 to 560 until the objective function is achieved. For example, the sequence image generator processor 150 and the sequence image discriminator processor 160 may have an adversarial relationship, wherein the sequence image generator processor 150 generates false / generated images and the sequence image discriminator processor 160 learns to distinguish between real sequence ultrasound images and generated sequence ultrasound images. Through competition and receiving feedback, the sequence image generator processor 150 may generate sequence ultrasound images similar to real sequence ultrasound images, and the sequence image discriminator processor 160 becomes better at distinguishing between real sequence ultrasound images and generated sequence ultrasound images. The goal is to make the generated sequence ultrasound images from the sequence image generator processor 150 similar to real ultrasound images and / or to make the sequence image discriminator processor 160 unable to distinguish between real ultrasound images and generated ultrasound images. Once the objective function is achieved, the training is completed at step 580.

[0063] Figure 6 600 is a flowchart showing exemplary steps 610 to 630 that may be used to enhance sequential ultrasound images using deep learning, according to various embodiments. Certain embodiments may omit one or more steps, and / or perform the steps in a different order than listed, and / or combine certain steps discussed below. For example, some steps may not be performed in certain embodiments. For another example, certain steps may be performed in a different time order than listed below, including simultaneously.

[0064] At step 610, the trained machine learning model may receive incomplete sequence image data, such as Figure 2As the ultrasound probe 104 slides over the region of interest of the anatomical structure and / or tissue and / or fluids such as blood contained therein, data may be collected by the signal processor 132. As the ultrasound probe 104 slides over the region of interest, the sequential image acquisition processor 140 collects incomplete sequential ultrasound image data. Since the image data is only available in one mode (e.g., Doppler mode), the image data may be incomplete, and the ultrasound system 100 may switch to other modes (e.g., B-mode or color Doppler mode) during intermittent periods.

[0065] At step 620, image data for the gap may be generated. The data may be degraded image data. The data may be generated by mixing noise (e.g., random noise) with image data obtained before the gap (e.g., immediately before the gap). The generated gap data and the incomplete data received at step 610 may be combined. The different data may be combined by the signal processor 132, or may be combined before being received by the signal processor 132 (e.g., by the sequence image generator processor 150).

[0066] At step 630, enhanced sequence image data may be generated by the signal processor 132. The enhanced sequence image data may be generated by the sequence image generator processor 150 based on previous training and combining incomplete data with image data placed in the gaps of the incomplete data. For example, the image data may be generated by the sequence image generator processor 150 based on previous training and combining incomplete data with image data placed in the gaps of the incomplete data. Figure 6 The sequence image generator processor 150 trained by the method of can receive incomplete image data and / or added image data (e.g., degraded image data) from the sequence image acquisition processor 140 and / or other sources, and can generate enhanced sequence ultrasound images. The enhanced sequence ultrasound image data generated by the signal processor 132 can be similar to the incomplete sequence image data, except that the enhanced sequence image data is complete and does not include gaps.

[0067] As used herein, the term "circuit" refers to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure hardware, is executed by hardware, and / or is otherwise associated with hardware. For example, as used herein, when executing one or more first codes, a specific processor and memory may include a first "circuit", and when executing one or more second codes, a specific processor and memory may include a second "circuit". As used herein, "and / or" represents any one or more of the items in the list connected by "and / or". For example, "x and / or y" represents any element in the three-element set {(x), (y), (x, y)}. As another example, "x, y and / or z" represents any element in the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. As used herein, the term "exemplary" means used as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "such as" lead to a list of one or more non-limiting examples, instances, or illustrations. As used herein, a circuit is “capable of operation” and / or “configured to” perform a function whenever the circuit includes the necessary hardware and code (if necessary) to perform the function, regardless of whether execution of the function is disabled or not enabled by some user-configurable setting.

[0068] Other embodiments may provide a computer-readable device and / or a non-transitory computer-readable medium, and / or a machine-readable device and / or a non-transitory machine-readable medium, on which are stored machine codes and / or a computer program having at least one code segment that can be executed by a machine and / or a computer, so that the machine and / or the computer performs the steps for enhancing sequential ultrasound images using deep learning as described herein.

[0069] Therefore, the present disclosure may be implemented in hardware, software, or a combination of hardware and software. The present disclosure may be implemented in a centralized manner in at least one computer system, or in a distributed manner, where different elements are distributed across several interconnected computer systems. Any type of computer system or other device suitable for executing the methods described herein is suitable.

[0070] The various embodiments may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein and which, when loaded into a computer system, is capable of carrying out these methods. A computer program herein refers to any expression of a set of instructions in any language, code or notation, which is intended to cause a system with information processing capabilities to perform specific functions directly or after either or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.

[0071] Although the present disclosure has been described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the specific embodiments disclosed, but the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A method for enhancing sequential ultrasound image data in an ultrasound system, the method comprising: acquiring, by an ultrasound probe of the ultrasound system, first image data during a first time period and acquiring third image data during a third time period after the first time period; as well as Inferring inferred image data corresponding to a second time period, wherein the second time period is between the first time period and the third time period, wherein the inferred image data is inferred by: feeding the degraded image data into a trained machine learning model; and The inferred image data is generated by the trained machine learning model.

2. The method according to claim 1, wherein: The first image data, the third image data, and the inferred image data include Doppler image data.

3. The method according to claim 2, wherein: The degraded image data is obtained by mixing noise with the first image data.

4. The method according to claim 2, wherein: The inferred image data includes time domain data.

5. The method according to claim 2, wherein: The extrapolated Doppler image data includes frequency domain data.

6. The method according to claim 2, wherein: The trained machine learning training model includes a generative adversarial network (GAN) model.

7. The method according to claim 6, wherein: The GAN model includes a generator and a discriminator, and the method further includes training the GAN model by: inputting the degraded image data into the generator; outputting, by the generator, samples corresponding to the degraded image data to the discriminator; inputting reference image data into the discriminator; as well as The degraded image data and the reference image data are discriminated using the discriminator according to at least one function.

8. The method according to claim 7, wherein: The degraded image data is generated by mixing noise with the first image data.

9. The method of claim 2, further comprising acquiring B-mode data during the second time period.

10. A system for enhancing sequential ultrasound image data, the system comprising: an ultrasound probe configured to obtain first image data during a first time period and to obtain third image data during a third time period after the first time period; and a processor configured to infer inferred image data corresponding to a second time period, wherein the second time period is between the first time period and the third time period, The inferred image data is inferred in the following manner: the degraded image data is input into a trained machine learning model; and the inferred image data is generated by the trained machine learning model.

11. The system according to claim 10, wherein: The first image data, the third image data, and the inferred image data include Doppler image data.

12. The system according to claim 11, wherein: The processor is further configured to generate the degraded image data by mixing noise with the first image data.

13. The system according to claim 10, wherein: The trained machine learning training model includes a generative adversarial network (GAN) model.

14. The system according to claim 13, wherein: The processor is further configured to implement the GAN model using a generator and a discriminator, and wherein the processor is further configured to train the GAN model by: inputting the degraded image data into the generator; outputting, by the generator, samples corresponding to the degraded image data to the discriminator; inputting reference image data into the discriminator; and The degraded image data and the reference image data are discriminated using the discriminator according to at least one function.

15. The system of claim 14, wherein: The degraded image data is generated by mixing noise with the first image data.

Images