Data processing device and data processing method

Abstract

To improve image quality.SOLUTION: A data processing device includes a processing circuit. The processing circuit inputs first ultrasound data to a neural network, calculates a first error by applying a first filter to the difference between second ultrasound data output from the neural network and target ultrasound data, calculates a second error by applying a second filter different from the first filter to the difference between the second ultrasound data and the target ultrasound data, calculates a loss value on the basis of the first error and the second error, and updates parameters of the neural network on the basis of the loss value to generate a trained neural network.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to a data processing device and a data processing method. [Background technology]

[0002] In ultrasound harmonic imaging, higher harmonics can provide images with significantly reduced artifacts, improved contrast-to-noise ratios, and improved lateral resolution. For some anatomical structures, such as blood vessels, higher-order harmonic images with improved contrast are desirable; for example, third-order harmonic images can provide images with fewer artifacts and improved resolution compared to second-order harmonic images.

[0003] Deep learning-based techniques offer a potential solution for improving harmonic imaging with superior image quality and fast image acquisition. To improve the accuracy of deep learning networks, common training methods involve increasing the complexity of the trained network (e.g., the depth or channels of the deep learning network). Therefore, deep learning-based methods for harmonic imaging can result in wide / deep networks with large memory and high computational power. The large memory and high computational cost can limit their applicability in practical use. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2023 / 0385643 Summary of the Invention [Problem to be solved by the invention]

[0005] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to improve image quality. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0006] A data processing device according to an embodiment includes a processing circuit that inputs first ultrasound data to a neural network, calculates a first error by applying a first filter to a difference between second ultrasound data output from the neural network and target ultrasound data, calculates a second error by applying a second filter different from the first filter to the difference between the second ultrasound data and the target ultrasound data, calculates a loss value based on the first error and the second error, and updates parameters of the neural network based on the loss value to generate a trained neural network. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a system diagram of an ultrasound imaging system according to one embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a method for ultrasound tissue harmonic imaging according to one embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram illustrating an ultrasound tissue harmonic imaging solution according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram illustrating training a neural network using a loss function according to one embodiment of the present disclosure. [Figure 5] FIG. 5 is a graph of an input spectrum, an output spectrum, and a high-pass filter according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram illustrating a weighted loss function according to one embodiment of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram illustrating training of a neural network based on features representing frequency, according to one embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart of a method for acquiring ultrasound data using a trained model according to one embodiment of the present disclosure. [Figure 9A] FIG. 9A is a flowchart of a method for acquiring ultrasound data using a trained model with features having frequency representations according to one embodiment of the present disclosure. [Figure 9B] FIG. 9B is a flowchart of a method for training a model to perform harmonic imaging using ultrasound signals according to one embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram of a hardware configuration of an apparatus for performing a method according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of a data processing device and a data processing method will be described in detail with reference to the drawings.

[0009] Many different embodiments or examples for implementing various features of the subject matter of the present disclosure are disclosed below. Specific examples of components and arrangements are described below to facilitate understanding of the present disclosure. Of course, these are merely examples and are not intended to limit the present disclosure. For example, in the following description, a reference to forming a first feature above or on a second feature may include an embodiment in which the first feature and the second feature are formed in direct contact with each other, or may include an embodiment in which an additional feature is formed between the first feature and the second feature such that the first feature and the second feature are not in direct contact with each other. Furthermore, reference numerals and / or letters may be repeated in various examples in the present disclosure. This repetition is for the purposes of brevity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations described. Furthermore, spatially relative terms such as “top,” “bottom,” “below,” “below,” “downward,” “above,” and the like may be used herein for ease of description to describe the relationship of one element or feature to another element or feature, as shown in the figures. Spatially relative terms are intended to encompass various orientations of the device during use or operation in addition to the orientation depicted in the figures. The system may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative terms used herein may be similarly interpreted accordingly.

[0010] The order of description of the various steps described herein is for clarity of description. In general, these steps can be performed in any suitable order. Furthermore, although each of the various features, techniques, configurations, etc. described herein is described in various places in this disclosure, it is contemplated that each concept can be practiced independently of one another or in combination. Accordingly, the present invention can be embodied and realized in various ways.

[0011] Ultrasound tissue harmonic imaging (THI) is a signal processing technique that irradiates internal tissue with an ultrasound beam and generates harmonics from nonlinear distortion during the transmit phase of the pulse-echo cycle. Tissue harmonic images are acquired by transmitting a frequency spectrum through a probe and receiving a frequency spectrum that includes fundamental echoes in a fundamental frequency band (referred to herein as the fundamental frequency) and harmonics generated within the body. Harmonic imaging acquires harmonic images by collecting harmonic signals generated by tissue and filtering the fundamental echo signals, resulting in clearer images. Harmonic signals (harmonics) are multiples of the fundamental frequency. Therefore, transmitting a frequency band centered at frequency f generates harmonic frequency bands centered at 2f, 3f, 4f, etc. (higher harmonics are referred to as second, third, fourth, etc.).

[0012] High-order harmonic imaging can produce images with fewer artifacts and improved resolution. However, high-order harmonic imaging (higher than the second harmonic) has limited penetration depth due to rapid attenuation of high-frequency signals. Second-order harmonic imaging can penetrate deeper than higher-order harmonics. However, second-order harmonic imaging suffers from many artifacts and a low contrast-to-noise ratio in the near field. For anatomical structures such as blood vessels, it is desirable to obtain high-order harmonic images with improved contrast.

[0013] In conventional ultrasound imaging, to acquire high-order harmonics, an ultrasound probe must sequentially transmit multiple pulses with opposite / desired phases to the same tissue. This results in a long image acquisition time and therefore a low frame rate. However, tissue motion can significantly affect image quality, so a high frame rate is especially important when such motion occurs. Furthermore, conventional ultrasound imaging for high-order harmonics is generally limited to the signal received by the probe. To focus on desired frequencies, filters may be used to filter out several frequency bands. However, filtering alone does not improve the image quality of ultrasound images.

[0014] To achieve harmonic imaging, three common methods for extracting harmonics from received echo signals are separation based on high-pass filtering, pulse inversion (PI), and power amplitude modulation (PAM). However, linear filtering suffers from spectral leakage where the fundamental and harmonic frequency bands overlap, and its performance depends on the cutoff frequency, filter order, and algorithm. PI and PAM overcome the aforementioned limitations, but they require two or more consecutive transmissions, resulting in a lower frame rate and susceptibility to motion artifacts, thus creating a trade-off between the achievable frame rate and the desired image quality.

[0015] Another example process is Differential Tissue Harmonic Imaging (DTHI). DTHI involves simultaneously transmitting two pulses at different frequencies, designated f1 and f2. In addition to these second-harmonic frequencies (2f1 and 2f2), other frequencies, the sum and difference of the transmitted frequencies (f2 + f1 and f2 - f1, respectively), are generated in the tissue. The second-harmonic signal at the lower frequency (2f1) and the difference frequency (f2 - f1), are detected by the probe. The other generated frequency components do not fit within the bandwidth of the probe. Higher resolution, better penetration, and fewer artifacts can be achieved with DTHI.

[0016] As described herein, deep learning-based techniques can provide solutions for improving harmonic imaging with superior image quality and fast acquisition. Deep learning-based harmonic imaging training can include low-quality, low-order inputs (e.g., IQ1 or IQ0, where IQ represents in-phase and quadrature-phase, or quadrature-demodulated data) and high-quality, high-order harmonic targets (e.g., IQ3 or denoised, clean IQ3). During training, simple multiplication coefficients or other representations of neural interconnections are iteratively adjusted to minimize the calculated overall error or loss between the network-processed input data and each corresponding target data. As described above, to improve accuracy, training methods can increase the complexity of the trained network (e.g., the depth or channels of the deep learning network). Therefore, deep learning-based methods for harmonic imaging can typically result in large / deep networks with large memory and high computational power.

[0017] Generally, to speed up performance, one can upgrade the hardware, perform quantization, or use other model structures, such as grouped convolutions, mobile nets, or networks with model scaling. However, these options may be costly, time-consuming, or require complex reorganization. Instead, this specification describes a training method that keeps the simple backbone of the network structure and leverages the frequency-based nature of ultrasound harmonic imaging to design a training method that emphasizes high-frequency components during training.

[0018] 1 is a system diagram of an ultrasound imaging system. The ultrasound imaging system 200 may include any of a series of probes 202 (ultrasound probes). Probe types include convex, linear, sector, and special purpose designs. Signals received by the probes 202 are processed by a computer system 204. The ultrasound imaging system 200 may also include multi-harmonic compounding, which combines signals from individual beams with overlapping data from adjacent beams.

[0019] Ultrasound images are created from ultrasound waves at frequencies above the range of human hearing, typically 1-10 MHz or higher. The probe 202 emits high-frequency waves and records the reflected waves (fundamental frequency) that bounce off tissue interfaces as a series of time-domain signals. One type of ultrasound image is an intensity image, also known as a B-mode image, which is a grayscale or brightness-based representation of an object.

[0020] The raw signals received by the probe 202 are in the radio frequency range and are known as radio frequency (RF) data. A series of signal processing steps are performed by the computer system 204 to convert the RF data into an ultrasound image, such as a B-mode image. One pre-processing step demodulates the RF data to baseband, downscaling the signal to reduce the bandwidth required to store the data. This new signal is called an in-phase and quadrature phase (IQ) signal and is typically represented as a complex number. In this disclosure, the terms IQ data and RF data are used interchangeably, as they both refer to the raw data from the probe, despite their different formats. Furthermore, deep neural network embodiments of the present disclosure can be configured to accept either IQ data or ultrasound images based on IQ data as input.

[0021] The computer system 204 may be integrated into a portable ultrasound device, may be a remote server, or may be a cloud service accessed via the internet. The ultrasound imaging system 200 may include at least one display device 206 for displaying one or more ultrasound images. The display device 206 may be an LCD display, an LED display, an organic LED display, or the like, having a display size and resolution sufficient to display the ultrasound images output by the computer system 204.

[0022] 2 shows an example of a method for ultrasonic tissue harmonic imaging according to an embodiment of the present disclosure. In order to acquire more signals in an ultrasonic far-field image, images can be obtained for both the second and third harmonics. As shown in FIG. 2, the ultrasonic imaging system 200 transmits ultrasonic waves at angles of 0°, 120°, 240°, and 180°, and obtains a reflected wave signal (IQ1) of "fundamental wave + third harmonic" by subtracting the reflected wave signal corresponding to the ultrasonic wave transmission at 0° from the reflected wave signal corresponding to the ultrasonic wave transmission at 180°. A second harmonic reflected wave signal (IQ2) can be obtained by adding the reflected wave signal corresponding to the 0° ultrasonic transmission with the reflected wave signal corresponding to the 180° ultrasonic transmission, and a third harmonic reflected wave signal (IQ3) can be obtained by adding the 0° reflected wave signal, the 120° reflected wave signal, and the 240° reflected wave signal. As mentioned above, one method, the DTHI method, simultaneously transmits two pulses at different frequencies, designated f1 and f2. In addition to these second harmonic frequencies (2f1 and 2f2), the sum and difference of the transmitted frequencies (f2 + f1 and f2 - f1, respectively) are also generated in the tissue. A second harmonic signal 316 at a lower frequency (2f1) and a second harmonic signal 318 at a difference frequency (f2 - f1) are detected by the probe 202. However, this method requires a long pulse generation interval 312. This results in slow frame rates and therefore poor quality imaging, especially when tissue is moving.

[0023] FIG. 3 illustrates a neural network-based method for ultrasound tissue harmonic imaging according to an embodiment of the present disclosure. In one embodiment, the imaging method generates pulses at short time intervals 412 by acquiring a signal that is a combination of a fundamental ultrasound frequency and a third harmonic 414 and simultaneously acquiring a second harmonic 416. A deep neural network 422 can generate an estimated third harmonic from the combined signal. The deep neural network 422 can be trained to effectively generate a denoised second harmonic from the resulting second harmonic. The deep neural network 422 can further be trained to effectively fuse the estimated third harmonic and the denoised second harmonic to generate an organ-specific fused image. That is, the final harmonic image 424 output by the deep neural network 422 can be a third harmonic image, a second harmonic image, or a fused image of the third harmonic image and the second harmonic image, etc.

[0024] The short time interval 412 for pulse generation allows for reduced motion effects and reduced artifacts. The deep neural network 422 can improve image quality with shorter signal acquisition times.

[0025] In one embodiment, the deep learning-based framework is configured to input various different combinations of IQ data received by the probe 202 and subjected to data processing steps in the computer system 204, and to output desired images with improved image quality, reduced near-field artifacts, improved contrast, and greater penetration depth. The deep learning-based framework can directly utilize second or third harmonics, or can use data including the fundamental frequency. The input IQ data can include a combination of a fundamental frequency signal, a second harmonic signal, and a third harmonic signal (e.g., IQ0). The IQ data can include a combination of a fundamental ultrasound frequency and a third harmonic (e.g., IQ1). The input IQ data can include only a second harmonic signal (e.g., IQ2) or only a third harmonic signal (e.g., IQ3). The input IQ data can also be other higher-order harmonics higher than third.

[0026] A deep learning-based framework can be subjected to a training method for a specific target, including desired harmonic data or desired types of images. The framework can be trained for feature recognition and depth dependency, and can be customized to account for patients with different body mass indexes (BMIs) and / or demographic information. For example, the deep learning-based framework being trained can be trained to output harmonic IQ data or a depth-dependent fusion map.

[0027] Deep learning-based frameworks can be configured as structures that include multi-layer perceptrons, convolutional neural networks such as U-net, or fusion networks. Convolutional neural networks (CNNs) have been used in visual recognition tasks. CNNs can be trained using large sets of training images, such as the 1 million training images in the ImageNet dataset. ImageNet has eight layers and millions of parameters. Very deep convolutional networks can be used for large-scale image recognition.

[0028] One deep learning network architecture (U-Net) can be trained with far fewer images. Training with thousands of images may be impossible for typical biomedical tasks. The U-Net architecture has an upsampling section and many feature channels, which allows the network to propagate contextual information to higher resolution layers. As a result, the original U-Net architecture consists of a reduction pass and an augmentation pass, with the augmentation pass being more or less symmetrical to the reduction pass, resulting in a U-shaped architecture.

[0029] The reduction pass follows the architecture of a typical convolutional neural network: it involves repeatedly applying two 3x3 convolutions (unpadded convolutions), each followed by a rectified linear unit (ReLU) and a 2x2 max-pooling operation with stride 2 for downsampling. At each downsampling step, the number of feature channels doubles.

[0030] Each step in the augmentation pass involves upsampling the feature map, followed by a 2x2 convolution (an "upconvolution") to halve the number of feature channels, concatenation with the corresponding cropped feature map from the reduction pass, and two 3x3 convolutions, each followed by a ReLU. The final layer uses a 1x1 convolution to map each of the 64-element feature vectors to the desired number of classes.

[0031] FIG. 4 is a schematic diagram of the use of a loss function in training, according to one embodiment of the present disclosure. In one embodiment, loss can be a measure of how well a deep learning model predicts a desired output from a given set of input targets. A loss function, also known as a cost function or objective function, can be configured to quantify the difference between the actual value and the value predicted (estimated) by the model. Thus, in the process of training a neural network, the loss function can be represented as "L." The parameter that controls the behavior of the system can be represented as θ. The gradient at which the loss function (L) penalizes errors as a function of θ can be used to improve the average agreement between the target and the neural network's output during training.

[0032] We use the variable k to index the training pairs and n to index the vector X k and Y k If we index the inputs in ∇ ...

[0033]

number

[0034] As described herein, the loss function L in equation (1) can take the form of equation (2) below:

[0035]

number

[0036] In the formula, S k denotes the error, and Φ is a suitable error metric that applies only in the passband of filter f1.

[0037] 5 is a graph illustrating an example spectrum for designing a filter according to an embodiment of the present disclosure. In one embodiment, the graph may include an input signal X (IQ1), a target signal Y (IQ3), and a designed filter f1 (IQ3-difference 1). Filter f1 may be a high-pass filter in the frequency domain, for example, to emphasize high frequencies for harmonic imaging during training.

[0038] In one embodiment, the first filter, filter f1, can be a high-pass or band-pass filter. Therefore, the band-pass or high-pass components can be emphasized in the resulting high-order harmonic data. Similarly, a weighted sum for the final total loss can be applied, penalizing both the overall difference between the network output and the target and the corresponding frequency components. This can be expressed as Equation (3) or Equation (4) below.

[0039]

number

[0040]

number

[0041] FIG. 6 is a schematic diagram of a method for training a network according to one embodiment of the present disclosure. In one embodiment, the second filter, function f2, can be another filter (e.g., an all-pass filter or a frequency band emphasis filter) or a B-mode calculation function. It should be noted that filters f1 and f2 can be designed based on a specific application. For example, the first filter, filter f1, can be a Sobel (operator or filter) or a Prewitt gradient edge detection filter. The Sobel operator performs a two-dimensional spatial gradient measurement on an image and can emphasize regions of high spatial frequency corresponding to edges. In general, a Sobel filter can be used to determine an approximate absolute gradient magnitude at each point of an input grayscale image. α can control the contribution from high-frequency components.

[0042] Returning to Figure 5, in one embodiment, the IQ3 curve may be the target during training. If only a single loss L1 is used, the single loss L1 does not emphasize the desired high frequency components. Therefore, a Sobel loss may be applied to IQ1 in Figure 5 to emphasize the high frequency components. Thus, the loss is a weighted sum. As explained in equation (3), the loss may be a weighted sum of the outputs of filter f1 and filter f2.

[0043] In one embodiment, filter f2 may be an all-pass filter, so that no specific frequency band is selected or targeted. A weight α may be applied to filter f1, as shown in equation (3), which may be a Sobel loss. The weights may also be carefully adjusted to obtain an optimally Sobel-weighted loss. Optimally Sobel-weighting the loss may result in improved imaging. The Sobel filter may be a two-dimensional filter that includes a pair of convolution kernels. The convolution kernels may be, for example, 3×3, 5×5, 7×7, etc.

[0044] As shown in FIG. 6, an input X (e.g., the IQ1 curve in FIG. 5) can be input to a network g(θ). The network can generate an inferred output, which can be compared with a target output Y (e.g., the IQ3 curve in FIG. 5) to calculate an error. That is, the error or difference between the inferred output and the target output Y can be filtered through a filter f2 to generate a second filtered error. Additionally or alternatively, the error or difference between the inferred output and the target output Y can be filtered through a filter f1 to generate a first filtered error. The first filtered error can additionally or alternatively be weighted by a weight α. The two filtered errors can be combined to generate an overall error, which is used to adjust the variable parameters of the network. Alternatively, filters f1 and f2 can output errors based on the inferred output and the target output Y, respectively. Filter f1 can be at least one of a high-pass filter and a band-pass filter. Filter f2 can be at least one of an all-pass filter and a frequency band emphasis filter. Thus, filter f1 is different from filter f2. The weight α controls the contribution from high frequency components.

[0045] As described above, in one embodiment, the deep learning-based framework can be configured to input various different combinations of IQ data received by the probe 202 and subjected to data processing steps in the computer system 204, and can be configured to output desired images with improved image quality, reduced near-field artifacts, improved contrast, and greater penetration depth. In one embodiment, the input data can be ultrasound images, such as intensity mode (B-mode) images. In one embodiment, the input data can be complex data having real and imaginary parts, such as IQ data. The deep learning-based framework can directly utilize second or third harmonics, or can use data including a fundamental frequency. The input IQ data, which is the first ultrasound data, can include a combination of a fundamental frequency signal, a second harmonic signal, and a third harmonic signal (IQ0). The input IQ data, which is the first ultrasound data, can include a combination of a fundamental ultrasound frequency and a third harmonic (IQ1). The input IQ data, which is the first ultrasound data, may include only the second harmonic signal (IQ2) or only the third harmonic signal (IQ3). The input IQ data may also include other higher-order harmonics higher than the third harmonic.

[0046] In one embodiment, a neural network model can be used to generate the output ultrasound data. For example, after sufficient training and adjustment, the input ultrasound data can be input to the neural network model to generate output ultrasound data that includes harmonic content.

[0047] FIG. 7 is a schematic diagram of training a neural network using features represented in the frequency domain, according to one embodiment of the present disclosure. In one embodiment, the frequency representation can be emphasized during training. Traditionally, the contribution of frequency components has been considered in loss function design, but not specifically emphasized. According to this training method, the frequency components of input data and target data are first analyzed and represented by a block function Ψ. The network can be trained to optimize the represented frequency features. During the inference phase, the output of the network can generate a frequency representation, and the block function Ψ can be inversely transformed from the frequency domain to the spatial domain.

[0048] Compared with upgrading hardware to speed up the processing time of large-scale models, the above-mentioned method is advantageous in that it is cost-effective and easy to implement. Compared with other methods that simplify the network structure model, the above-mentioned method can use a simple U-net as the backbone network with fewer coefficients while maintaining high inference accuracy, better SNR ratio, and better image contrast.

[0049] The above-described method for designing the loss function is flexible for various applications and will not increase the inference time while maintaining sufficient performance. Also, the above-described method can maintain a high frame rate in real-time imaging, avoid the problem of motion artifacts, or avoid the spectral leakage in conventional harmonic imaging.

[0050] FIG. 8 illustrates, by way of non-limiting example, a flowchart of a method 800 for acquiring ultrasound data using a trained model, according to one embodiment of the present disclosure.

[0051] In one embodiment, step S805 includes acquiring first ultrasound data, which may include, for example, a fundamental component and a harmonic component.

[0052] In one embodiment, step S810 includes a processing circuit (e.g., a processing circuit included in device 601 shown in FIG. 10) inputting the first ultrasound data into a model or neural network. The model or neural network may be configured to generate second ultrasound data as an output. That is, the second ultrasound data is output from the neural network. The second output ultrasound data may be, for example, a specific frequency component of the input first ultrasound data.

[0053] In one embodiment, step S815 includes acquiring second ultrasound data, which may include, for example, harmonic components.

[0054] In one embodiment, step S820 includes the processing circuit calculating a first error by applying a first filter to the difference between the second ultrasound data output from the neural network and the target ultrasound data.

[0055] In one embodiment, step S825 includes the processing circuit calculating a second error by applying a second filter to the difference between the second ultrasound data and the target ultrasound data, where the second filter may be different from the first filter.

[0056] In one embodiment, step S830 includes the processing circuit weighting the first error or the second error, or both the first error and the second error.

[0057] In one embodiment, step S835 includes the processing circuitry calculating a loss value based on the calculated first error and the calculated second error. As described above, the processing circuitry can weight the first error when calculating the loss value. As described above, the processing circuitry can weight the second error in combination with the first error. Alternatively, the processing circuitry can weight the second error without weighting the first error.

[0058] In one embodiment, step S840 includes the processing circuit updating parameters of the neural network based on the calculated loss value to generate a trained model (neural network).

[0059] In summary, the data processing device according to the embodiment includes a processing circuit that inputs first ultrasound data to a neural network, calculates a first error by applying a first filter to the difference between second ultrasound data output from the neural network and target ultrasound data, calculates a second error by applying a second filter different from the first filter to the difference between the second ultrasound data and the target ultrasound data, calculates a loss value based on the first error and the second error, and updates the parameters of the neural network based on the loss value to generate a trained neural network.

[0060] Furthermore, the data processing method according to the embodiment includes the steps of inputting first ultrasonic data to a neural network, calculating a first error by applying a first filter to the difference between second ultrasonic data output from the neural network and the target ultrasonic data, calculating a second error by applying a second filter different from the first filter to the difference between the second ultrasonic data and the target ultrasonic data, calculating a loss value based on the first error and the second error, and updating the parameters of the neural network based on the loss value to generate a trained neural network.

[0061] In one embodiment, in step S843, the method 800 may be repeated again to step S840.

[0062] In one embodiment, step S845 includes acquiring third ultrasound data using the trained model with updated parameters. The third ultrasound data may be, for example, an improved ultrasound image. That is, in step S845, inference is performed using the neural network of the embodiment.

[0063] That is, the data processing device according to the embodiment includes a processing circuit that inputs ultrasound data obtained by performing an ultrasound scan on a subject via an ultrasound probe into a trained neural network and outputs the ultrasound data, and the trained neural network is generated by inputting input ultrasound data into the neural network, calculating a first error by applying a first filter to the difference between output ultrasound data output from the neural network and target ultrasound data, calculating a second error by applying a second filter different from the first filter to the difference between the output ultrasound data and the target ultrasound data, calculating a loss value based on the first error and the second error, and updating the parameters of the neural network based on the loss value.

[0064] FIG. 9A illustrates, by way of non-limiting example, a flowchart of a method 900 for acquiring ultrasound data using a trained model with features having frequency representations, according to one embodiment of the present disclosure.

[0065] In one embodiment, step S905 includes acquiring first ultrasound data, which may include, for example, a fundamental component and a harmonic component.

[0066] In one embodiment, step S910 includes converting the first ultrasound data into features having a frequency representation if the first ultrasound data is not already in a frequency representation.

[0067] In one embodiment, step S915 includes the processing circuit inputting the first ultrasound data in the form of a frequency representation into a model or neural network. The model or neural network may be configured to generate second ultrasound data as output, also in the form of a frequency representation. The output second ultrasound data may be, for example, a specific frequency component of the input first ultrasound data.

[0068] In one embodiment, step S920 includes the processing circuit acquiring second ultrasound data. The second ultrasound data may include, for example, harmonic components. The second ultrasound data may be in the form of a frequency representation.

[0069] In one embodiment, step S925 includes calculating a first error by applying a first filter to the difference between the second ultrasound data and the target ultrasound data, which may be in the form of a frequency representation.

[0070] In one embodiment, step S930 includes the processing circuit calculating a second error by applying a second filter to the difference between the second ultrasound data and the target ultrasound data, the target ultrasound data being in the form of a frequency representation.

[0071] In one embodiment, step S935 includes the processing circuit weighting the first error or the second error, or both the first error and the second error.

[0072] In one embodiment, step S940 includes the processing circuitry calculating a loss value based on the first error and the second error. As described above, the first error can be weighted. As described above, the second error can be weighted in combination with the first error. Alternatively, the first error can be unweighted and the second error can be weighted. Alternatively, the processing circuitry can weight the first error to adjust for contributions from high frequency components.

[0073] In one embodiment, step S945 includes updating parameters of the model based on the calculated loss value to generate a trained model.

[0074] In one embodiment, in step S948, the method 900 may be repeated again up to step S945.

[0075] In one embodiment, step S950 includes acquiring third ultrasound data using the trained model with the updated parameters, i.e., in step S950, inference is performed using the neural network of the embodiment.

[0076] FIG. 9B illustrates, by way of example and not limitation, a flowchart of a method 901 for implementing (inferring) a model for performing harmonic imaging using ultrasound signals, according to an embodiment of the present disclosure.

[0077] In one embodiment, step S955 includes acquiring first ultrasound data.

[0078] In one embodiment, step S960 includes inputting the acquired first ultrasound data into a trained neural network to acquire second ultrasound data. The trained neural network is trained using the training input data, corresponding target data, and a loss function. The loss function calculates a first error by applying a first filter to a difference between output ultrasound data output from the neural network in response to the input of the training input data and the target ultrasound data, calculates a second error by applying a second filter different from the first filter to the difference between the output ultrasound data and the target ultrasound data, calculates a loss value using the loss function that is a function of the calculated first error and the calculated second error, and updates parameters of the trained neural network based on the calculated loss value to calculate the loss value.

[0079] In one embodiment, step S965 includes outputting the second ultrasound data for display on a display.

[0080] Next, hardware of device 601 according to an exemplary embodiment will be described with reference to FIG. 10. In FIG. 10, device 601, which may be the processing device described above, is an example of a data processing device according to an embodiment and includes a processing circuit as described above. The processing circuit includes one or more elements described below with reference to FIG. 10. Device 601 may also include other components not explicitly shown in FIG. 10, such as a central processing unit (CPU), a graphical processing unit (GPU), a frame buffer, etc. In FIG. 10, device 601 includes a CPU 600 that executes the above / below-described processes. Process data and instructions may be stored in memory 602. These processes and instructions may be stored on a storage medium disk 604, such as a hard drive (HDD) or a portable storage medium, or may be stored remotely. Furthermore, the claimed advancement is not limited by the form of a computer-readable medium that stores instructions for the processes of the present invention. For example, the instructions may be stored on a CD (Compact Disc), a DVD (Digital Versatile Disk), FLASH (registered trademark) memory, RAM (Random Access Memory), ROM (Read Only Memory), PROM (registered trademark), EPROM (registered trademark), EEPROM (registered trademark), a hard disk, or any other information processing device with which device 601 communicates, such as a server or computer.

[0081] Additionally, the claimed advancements may be provided as a utility application, background daemon, or operating system component, or a combination thereof, and run in conjunction with CPU 600 and an operating system such as Microsoft® Windows®, UNIX®, Solaris®, LINUX®, Apple®, MAC-OS®, and other systems known to those skilled in the art.

[0082] The hardware elements for implementing device 601 may be implemented using various circuit elements known to those skilled in the art. For example, CPU 600 may be an Intel® Xenon® or Core processor, or an AMD® Opteron® processor, or other types of processors recognized by those skilled in the art. Alternatively, CPU 600 may be implemented in a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or a Programmable Logic Device (PLD), or may be implemented using discrete logic circuitry, as recognized by those skilled in the art. Furthermore, CPU 600 may be implemented as multiple processors operating cooperatively in parallel to perform the processes described above.

[0083] The device 601 of FIG. 10 includes a network controller 606, such as an Intel® Ethernet® PRO network interface card from Intel Corporation of America, that interfaces with a network 650 to communicate with other devices. It will be appreciated that the network 650 may be a public network, such as the Internet, a private network, such as a LAN or WAN network, or any combination thereof, and may include a PSTN or ISDN subnetwork. The network 650 may also be a wired network, such as an Ethernet network, or a wireless network, such as a cellular network, including EDGE, 3G, 4G, and 5G wireless cellular systems. The wireless network may also be Wi-Fi, Bluetooth, or any other known form of wireless communication.

[0084] The device 601 further includes a display controller 608, such as an NVIDIA® GeForce® GTX or Quadro® graphics adapter from NVIDIA® Corporation, USA, for use with a display 610, such as an LCD monitor. A general-purpose I / O interface 612 interfaces with a keyboard and / or mouse 614 and a touchscreen panel 616, either on the display 610 or separate from the display 610. The general-purpose I / O interface also connects to various peripheral devices 618, including printers and scanners.

[0085] A sound controller 620 is also provided in the device 601 and interfaces with a speaker / microphone 622 to provide sound and / or music.

[0086] A generic storage controller 624 connects the storage media disks 604 to a communication bus 626 that interconnects all components of the device 601. The communication bus 626 may be ISA, EISA, VESA, PCI, or the like. Descriptions of the general features and functionality of the display 610, keyboard and / or mouse 614, display controller 608, storage controller 624, network controller 606, sound controller 620, and generic I / O interface 612 are omitted herein for the sake of brevity, as these features are known.

[0087] Specific details have been described above, such as the specific configuration of the processing system and the various components and processes used in the processing system. However, it should be understood that the technology described herein may be practiced in other embodiments that deviate from these specific details, and that such details are for purposes of explanation and not limitation. The embodiments disclosed herein have been described with reference to the accompanying drawings. Similarly, for purposes of explanation, specific numbers, materials, and configurations have been set forth to provide a thorough understanding. However, embodiments may be practiced without such specific details. Components having substantially the same functional structure may be designated by the same reference numerals, thereby avoiding redundant description.

[0088] To facilitate understanding of the various embodiments, various techniques have been described as a number of separate operations. The order of description should not be construed to imply that these operations are necessarily order dependent. In fact, these operations need not be performed in the order presented. The described operations may also be performed in a different order than in the described embodiments. In further embodiments, various additional operations may be performed and / or described operations may be omitted.

[0089] Embodiments of the present disclosure may be as described in the following appendices.

[0090] (1) An apparatus for training a model that performs harmonic imaging using ultrasound signals, comprising a processing circuit configured to input first ultrasound data to a neural network model configured to generate and output second ultrasound data, calculate a first error by applying a first filter to a difference between the second ultrasound data and target ultrasound data, calculate a second error by applying a second filter different from the first filter to the difference between the second ultrasound data and the target ultrasound data, calculate a loss value based on the calculated first error and the calculated second error, and update parameters of the neural network model based on the calculated loss value to generate a trained neural network model.

[0091] (2) The apparatus of (1), wherein the processing circuitry is further configured to weight the first error when calculating the loss value.

[0092] (3) The device described in either (1) or (2), wherein the processing circuit is further configured to apply the first filter, which is a high-pass filter or a band-pass filter, to the difference between the second ultrasound data and the target ultrasound data.

[0093] (4) The device described in any one of (1) to (3), wherein the processing circuit is further configured to apply the first filter, which is a Sobel filter, to the difference between the second ultrasound data and the target ultrasound data.

[0094] (5) The device described in any one of (1) to (4), wherein the processing circuit is further configured to apply the second filter, which is an all-pass filter, to the difference between the second ultrasound data and the target ultrasound data.

[0095] (6) The device described in any one of (1) to (5), wherein the processing circuit is further configured to input the first ultrasound data including a fundamental frequency component and a third harmonic component to the neural network model.

[0096] (7) The device described in any one of (1) to (6), wherein the processing circuit is further configured to convert the first ultrasound data into features having a frequency representation before inputting the first ultrasound data into the neural network model.

[0097] (8) The device described in any one of (1) to (7), wherein the processing circuit is further configured to acquire the second ultrasound data in the form of the frequency representation.

[0098] (9) The device described in any one of (1) to (8), wherein the processing circuit is further configured to calculate the first error and the second error using the target ultrasound data in the form of the frequency representation.

[0099] (10) The device according to any one of (1) to (9), wherein the processing circuit is further configured to weight the first error to adjust contributions from high frequency components.

[0100] (11) A method for training a model that performs harmonic imaging using ultrasound signals, the method comprising: inputting first ultrasound data to a neural network model configured to generate and output second ultrasound data; calculating a first error by applying a first filter to a difference between the second ultrasound data and target ultrasound data; calculating a second error by applying a second filter different from the first filter to the difference between the second ultrasound data and the target ultrasound data; calculating a loss value based on the calculated first error and the calculated second error; and updating parameters of the neural network model based on the calculated loss value to generate a trained neural network model.

[0101] (12) The method according to (11), wherein the step of calculating the loss value further includes weighting the first error.

[0102] (13) The method according to either (11) or (12), wherein the first filter is a high-pass filter or a band-pass filter.

[0103] (14) The method according to any one of (11) to (13), wherein the first filter is a Sobel filter.

[0104] (15) The method according to any one of (11) to (14), wherein the second filter is an all-pass filter.

[0105] (16) The method according to any one of (11) to (15), wherein the first ultrasound data includes a fundamental frequency component and a third harmonic component.

[0106] (17) The method according to any one of (11) to (16), further comprising converting the first ultrasound data into features having a frequency representation before inputting the first ultrasound data into the neural network model.

[0107] (18) A method according to any one of (11) to (17), further comprising the step of acquiring the second ultrasound data in the form of the frequency representation.

[0108] (19) The method according to any one of (11) to (18), further comprising a step of calculating the first error and the second error using the target ultrasound data in the form of the frequency representation.

[0109] (20) An apparatus for performing harmonic imaging using ultrasound signals, comprising a processing circuit configured to: acquire first ultrasound data; input the acquired first ultrasound data to a trained neural network to acquire second ultrasound data; the trained neural network is trained using training input data, corresponding target data, and a loss function; the loss function is configured to: calculate a first error by applying a first filter to a difference between the output ultrasound data output from the neural network in response to input of the training input data and the target ultrasound data; calculate a second error by applying a second filter different from the first filter to the difference between the output ultrasound data and the target ultrasound data; calculate a loss value using the loss function, which is a function of the calculated first error and the calculated second error; calculate the loss value by updating parameters of the trained neural network based on the calculated loss value; and output the second ultrasound data for display on a display.

[0110] According to at least one of the embodiments described above, image quality can be improved.

[0111] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0112] 200 Ultrasound Imaging System 202 Probe 204 Computer Systems 206 Display device 601 Equipment 602 memory

Claims

1. inputting the first ultrasound data into the neural network; calculating a first error by applying a first filter to a difference between the second ultrasonic data output from the neural network and the target ultrasonic data; calculating a second error by applying a second filter different from the first filter to a difference between the second ultrasound data and the target ultrasound data; calculating a loss value based on the first error and the second error; a processing circuit for updating parameters of the neural network based on the loss value to generate a trained neural network;

2. The data processing device according to claim 1 , wherein the processing circuit weights the first error when calculating the loss value.

3. 2. The data processing apparatus according to claim 1, wherein the first filter is a high-pass filter or a band-pass filter.

4. 2. The data processing apparatus of claim 1, wherein the first filter is a Sobel filter.

5. 2. The data processing apparatus according to claim 1, wherein the second filter is an all-pass filter.

6. The data processing apparatus according to claim 1 , wherein the first ultrasound data includes a fundamental frequency component and a third harmonic component.

7. The data processing apparatus according to claim 1 , wherein the first ultrasound data is data in the form of a frequency representation.

8. 8. The data processing apparatus of claim 7, wherein the second ultrasound data is in the form of a frequency representation.

9. 8. The data processing apparatus of claim 7, wherein the target ultrasound data is in the form of a frequency representation.

10. 3. The data processing apparatus according to claim 2, wherein said processing circuitry applies said weighting to said first error to adjust for contributions from high frequency components.

11. inputting the first ultrasound data into the neural network; calculating a first error by applying a first filter to a difference between the second ultrasonic data output from the neural network and the target ultrasonic data; calculating a second error by applying a second filter different from the first filter to a difference between the second ultrasound data and the target ultrasound data; calculating a loss value based on the first error and the second error; updating parameters of the neural network based on the loss value to generate a trained neural network.

12. a processing circuit for inputting first ultrasound data obtained by performing an ultrasound scan on a subject via an ultrasound probe into the trained neural network and outputting second ultrasound data; The trained neural network Input ultrasound data into the neural network, Calculating a first error by applying a first filter to a difference between the output ultrasound data output from the neural network and the target ultrasound data; calculating a second error by applying a second filter different from the first filter to a difference between the output ultrasonic data and the target ultrasonic data; calculating a loss value based on the first error and the second error; and updating parameters of the neural network based on the loss value.

Images