Systems and methods for improved panoramic ultrasound images

Abstract

Systems (100) and methods (600, 700) for: acquiring ultrasound image data of an anatomical structure by an ultrasound probe (104) of an ultrasound system (100); generating (606, 704), by at least one processor (132, 140) of the ultrasound system (100), an acquired panoramic ultrasound image (300) of the anatomical structure based on the ultrasound image data, the acquired panoramic ultrasound image (300) comprising a loss area (310, 410); processing (608, 706), by at least one processor (132, 150) of the ultrasound system (100), the acquired panoramic ultrasound image (300) based on one or more reference panoramic ultrasound images (400) without the lost area (310, 410, 520); and outputting, by at least one processor (132, 150) of the ultrasound system (100), the enhanced panoramic ultrasound image without the lost area (310, 410, 530) for presentation on a display system (134).

Description

Technical Field

[0001] Certain embodiments relate to ultrasound imaging. More specifically, certain embodiments relate to methods and systems for enhancing panoramic ultrasound images. Background Art

[0002] Ultrasound imaging is a medical imaging technique used to image organs and soft tissues in the human body. Ultrasound imaging uses real-time, non-invasive, high-frequency sound waves to produce two-dimensional (2D), three-dimensional (3D), and / or four-dimensional (4D) (i.e., real-time / continuous 3D images) images.

[0003] Panoramic ultrasound imaging is a powerful tool for obtaining images of organs and soft tissues that are larger than the field of view by stitching images together. Panoramic ultrasound images are obtained by sliding the ultrasound probe over the surface of the organ or soft tissue and adding the image data to the image data already collected.

[0004] However, a panoramic ultrasound image may sometimes contain a loss area due to the natural curvature of the human body. In order to try to capture a smaller loss area, a panoramic ultrasound image may be acquired again, or a panoramic ultrasound image with a loss area may be used to attempt a diagnosis.

[0005] Further limitations and disadvantages of conventional and traditional approaches will become apparent to those skilled in the art by comparing such systems with certain aspects of the present disclosure as set forth in the remainder of this application with reference to the accompanying figures. Summary of the invention

[0006] A system and / or method for enhancing panoramic ultrasound images using a deep learning approach is provided, the system and / or method substantially as shown and / or described in conjunction with at least one of the accompanying drawings, as more fully set forth in the claims.

[0007] These and other advantages, aspects and novel features of the present disclosure, as well as details of illustrated embodiments thereof, will be more fully understood from the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram 100 of an exemplary ultrasound system operable to enhance panoramic ultrasound images using deep learning, in accordance with various embodiments.

[0009] Figure 2 is an exemplary two-dimensional (2D) panoramic ultrasound image 300 including an area of ​​loss, according to various embodiments.

[0010] Figure 3 is an exemplary two-dimensional (2D) reference panoramic ultrasound image 400 according to various embodiments.

[0011] Figure 4 is an exemplary flow chart 500 of an exemplary deep learning technique according to various embodiments.

[0012] Figure 5 600 illustrating exemplary steps that may be used to train deep learning techniques to enhance panoramic ultrasound images, according to various embodiments.

[0013] Figure 6 is a flowchart 700 illustrating exemplary steps that may be used to enhance panoramic ultrasound images using deep learning, according to various embodiments. DETAILED DESCRIPTION

[0014] Certain embodiments may be found in methods and systems for enhancing panoramic ultrasound images using deep learning. Aspects of the present disclosure have the following technical effects, namely, panoramic ultrasound images are enhanced using deep learning to help provide a diagnosis. Various embodiments have the following technical effects, namely, processing acquired ultrasound images to enhance panoramic ultrasound images using deep learning. Certain embodiments have the following technical effects, namely, processing a degraded panoramic ultrasound image of a region of interest with a lost area and a reference ultrasound image of the same region of interest without a lost area. Aspects of the present disclosure have the following technical effects, namely, generating an improved ultrasound panoramic image using an ultrasound reference image. Various embodiments have the following technical effects, namely, processing an ultrasound image with a lost area to restore the lost area using a deep learning method, so as to provide a more complete panoramic image to a user to aid in diagnosis.

[0015] When read in conjunction with the accompanying drawings, the following specific embodiments of the foregoing invention content and certain embodiments will be better understood. In terms of the scope of the figures of the functional blocks of each embodiment shown in the accompanying drawings, these functional blocks do not necessarily represent the division between hardware circuits. Therefore, for example, one or more functional blocks (e.g., processors or memories) can be implemented in a single piece of hardware (e.g., a general signal processor or random access memory block, a hard disk, etc.) or multiple pieces of hardware. Similarly, a program can be a stand-alone program, can be incorporated into an operating system as a subroutine, can be a function in an installed software package, etc. It should be understood that each embodiment is not limited to the arrangements and tools shown in the accompanying drawings. It should also be understood that the embodiments can be combined, or other embodiments can be utilized, and structural, logical and electrical changes can be made without departing from the scope of each embodiment. Therefore, the following detailed description should not be regarded as a restrictive meaning, and the scope of the present disclosure is limited by the attached claims and their equivalents.

[0016] As used herein, elements or steps listed in the singular and beginning with the word "one" or "an" should be understood as not excluding a plurality of said elements or steps, unless such exclusion is explicitly stated. In addition, references to "exemplary embodiments," "various embodiments," "certain embodiments," "representative embodiments," etc. are not intended to be interpreted as excluding the presence of additional embodiments that also include the features of the narration. In addition, unless explicitly stated to the contrary, embodiments that "comprise," "include," or "have" an element or multiple elements having a particular attribute may include additional elements that do not have that attribute.

[0017] In addition, as used herein, the term "image" refers broadly to both visible images and data representing visible images. However, many embodiments generate (or are configured to generate) at least one visible image. In addition, as used herein, the phrase "image" is used to refer to an ultrasound mode, which can be one-dimensional (1D), two-dimensional (2D), three-dimensional (3D) or four-dimensional (4D), and includes brightness mode (B mode), motion mode (M mode), color motion mode (CM mode), color flow mode (CF mode), pulsed wave (PW) Doppler, continuous wave (CW) Doppler, contrast enhanced ultrasound (CEUS) and / or B mode and / or CF mode sub-mode, such as harmonic imaging, shear wave elastic imaging (SWEI), strain elastic imaging, tissue velocity imaging (TVI), power Doppler imaging (PDI), B flow, microangiography (MVI), ultrasound-guided attenuation parameters (UGAP), etc.

[0018] Furthermore, as used herein, the term processor or processing unit refers to any type of processing unit that can perform the required computations required by various embodiments, such as a single-core or multi-core CPU, an accelerated processing unit (APU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or a combination thereof.

[0019] It should be noted that various embodiments of generating or forming an image described herein may include a process for forming an image that includes beamforming in some embodiments and does not include beamforming in other embodiments. For example, an image may be formed without beamforming, such as by multiplying a matrix of demodulated data by a matrix of coefficients such that the product is an image and wherein the process does not form any "beams." Additionally, the formation of an image may be performed using a combination of channels that may originate from more than one transmit event (e.g., synthetic aperture techniques).

[0020] In various embodiments, ultrasound processing to form images, including ultrasound beamforming, such as receive beamforming, is performed, for example, in software, firmware, hardware, or a combination thereof. One specific implementation of an ultrasound system having a software beamformer architecture formed according to various embodiments is described in Figure 1 Shown in.

[0021] Figure 1 is a block diagram of an exemplary ultrasound system operable to automatically place a medical device in an anatomical structure using a locking mechanism in accordance with various embodiments. Figure 1 , an ultrasound system 100 and a training system 200 are shown. The ultrasound system 100 includes a transmitter 102, an ultrasound probe 104, a transmit beamformer 110, a receiver 118, a receive beamformer 120, an analog-to-digital (A / D) converter 122, a radio frequency (RF) processor 124, a radio frequency quadrature (RF / IQ) buffer 126, a user input device 130, a signal processor 132, an image buffer 136, a display system 134, and an archive 138.

[0022] The transmitter 102 may include suitable logic components, circuits, interfaces and / or codes that are operable to drive the ultrasound probe 104. The ultrasound probe 104 may be a linear, convex, intracavitary or phased array transducer. The ultrasound probe 104 may include a two-dimensional (2D) array of piezoelectric elements. The ultrasound probe 104 may include a group of transmitting transducer elements 106 and a group of receiving transducer elements 108 that generally constitute the same element. The group of transmitting transducer elements 106 may emit an ultrasound signal that passes through the oil and the probe cap and enters the target. In a representative embodiment, the ultrasound probe 104 is operable to collect ultrasound image data covering at least a large portion of an anatomical structure (such as a heart, an ovary or any suitable anatomical structure). In an exemplary embodiment, the ultrasound probe 104 may be operated in a volume acquisition mode, wherein the transducer assembly of the ultrasound probe 104 acquires a plurality of parallel 2D ultrasound slices that form an ultrasound volume.

[0023] The transmit beamformer 110 may include suitable logic, circuitry, interfaces, and / or code operable to control the transmitter 102 to drive the set of transmit transducer elements 106 via the transmit sub-aperture beamformer 114 to transmit ultrasound transmit signals into a region of interest (e.g., a person, an animal, an underground cavity, a physical structure, etc.). The transmitted ultrasound signals may be backscattered from structures in the object of interest (such as blood cells or tissue) to generate echoes. The echoes are received by the receive transducer elements 108.

[0024] The set of receive transducer elements 108 in the ultrasound probe 104 may be operable to convert received echoes into analog signals, sub-aperture beamformed by the receive sub-aperture beamformer 116, and then transmitted to the receiver 118. The receiver 118 may include suitable logic, circuitry, interfaces, and / or code that may be operable to receive the signals from the receive sub-aperture beamformer 116. The analog signals may be transmitted to one or more of the plurality of A / D converters 122.

[0025] The plurality of A / D converters 122 may include suitable logic components, circuits and interfaces and / or codes that are operable to convert analog signals from the receiver 118 into corresponding digital signals. The plurality of A / D converters 122 are disposed between the receiver 118 and the RF processor 124. Nevertheless, the present disclosure is not limited in this regard. Therefore, in some embodiments, the plurality of A / D converters 122 may be integrated within the receiver 118.

[0026] The RF processor 124 may include suitable logic components, circuits, interfaces, and / or codes that are operable to demodulate the digital signals output by the plurality of A / D converters 122. According to an embodiment, the RF processor 124 may include a complex demodulator (not shown) that is operable to demodulate the digital signals to form I / Q data pairs representing corresponding echo signals. The RF or I / Q signal data may then be transferred to the RF / IQ buffer 126. The RF / IQ buffer 126 may include suitable logic components, circuits, interfaces, and / or codes that are operable to provide temporary storage of the RF or I / Q signal data generated by the RF processor 124.

[0027] The receive beamformer 120 may comprise suitable logic, circuitry, interfaces, and / or code operable to perform digital beamforming processing, for example, to sum delayed channel signals received from the RF processor 124 via the RF / IQ buffer 126 and output a beam summed signal. The resulting processed information may be a beam summed signal output from the receive beamformer 120 and communicated to the signal processor 132. According to some embodiments, the receiver 118, the plurality of A / D converters 122, the RF processor 124, and the beamformer 120 may be integrated into a single beamformer, which may be a digital beamformer. In various embodiments, the ultrasound system 100 comprises a plurality of receive beamformers 120.

[0028] The user input device 130 may be used to enter patient data; scan parameters; settings; select a protocol and / or template; select a target structure for image acquisition; enter and / or select a region of interest; modify a region of interest; select a region of interest for image acquisition, focus / zoom volume; etc. In an exemplary embodiment, the user input device 130 may be operable to configure, manage, and / or control the operation of one or more components and / or modules in the ultrasound system 100. In this regard, the user input device 130 may be operable to configure, manage, and / or control the operation of the transmitter 102, ultrasound probe 104, transmit beamformer 110, receiver 118, receive beamformer 120, RF processor 124, RF / IQ buffer 126, user input device 130, signal processor 132, image buffer 136, display system 134, and / or archive 138. The user input device 130 may include buttons, rotary encoders, touch screens, motion tracking, voice recognition, a mouse device, a keyboard, a camera, and / or any other device capable of receiving user instructions. In certain embodiments, one or more of the user input devices 130 may be integrated into other components, such as the display system 134 or the ultrasound probe 104, for example. For example, the user input device 130 may include a touch screen display.

[0029] The signal processor 132 may include suitable logic components, circuits, interfaces and / or codes that are operable to process ultrasound scan data (i.e., summed IQ signals) to generate ultrasound images for presentation on a display system 134. The signal processor 132 is operable to perform one or more processing operations based on a plurality of selectable ultrasound modalities on the acquired ultrasound scan data. In an exemplary embodiment, the signal processor 132 may be used to perform display processing and / or control processing, etc. The acquired ultrasound scan data may be processed in real time during a scanning session as echo signals are received. Additionally or alternatively, the ultrasound scan data may be temporarily stored in the RF / IQ buffer 126 during a scanning session and processed in a less real-time manner in an online operation or an offline operation. In various embodiments, the processed image data may be presented at the display system 134 and / or may be stored at the archive 138. The archive 138 may be a local archive, a picture archiving and communication system (PACS), or any suitable device for storing images and related information.

[0030] The signal processor 132 may be one or more central processing units, microprocessors, microcontrollers, etc. For example, the signal processor 132 may be an integrated component, or may be distributed in various locations. In an exemplary embodiment, the signal processor 132 may include a panoramic image acquisition processor 140, a panoramic image generator processor 150, and a panoramic image discriminator processor 160. The signal processor 132 may be capable of receiving input information from the user input device 130 and / or the archive 138, generating output that may be displayed by the display system 134, and manipulating the output in response to the input information from the user input device 130, etc. For example, the signal processor 132, the panoramic image acquisition processor 140, and the panoramic image generator processor 150 may be capable of executing any of the methods and / or instruction sets discussed herein according to various embodiments.

[0031] The ultrasound system 100 may be operable to continuously acquire ultrasound scan data at a frame rate suitable for the imaging situation under consideration. Typical frame rates are in the range of 20 to 120, but may be lower or higher. The acquired ultrasound scan data may be displayed on the display system 134 at a display rate that is the same as the frame rate, or at a display rate that is slower or faster than the frame rate. An image buffer 136 is included to store processed frames of acquired ultrasound scan data that are not scheduled for immediate display. Preferably, the image buffer 136 has sufficient capacity to store ultrasound scan data frames equivalent to at least several minutes. The frames of ultrasound scan data are stored in a manner that is easy to retrieve from them according to their acquisition order or time. The image buffer 136 may be embodied as any known data storage medium.

[0032] The signal processor 132 may include a panoramic image acquisition processor 140 including suitable logic components, circuitry, interfaces, and / or code operable to obtain a panoramic ultrasound image using the ultrasound probe 104. In an exemplary embodiment, the panoramic image acquisition processor 140 may be configured to capture a panoramic ultrasound image and / or ultrasound image slices to detect and locate a target structure. For example, the panoramic image acquisition processor 140 may be configured to receive a user input selecting a target structure before performing ultrasound image acquisition and analyzing ultrasound images and / or volumes of the ultrasound image acquisition to detect and locate the target structure.

[0033] Figure 23 is an exemplary two-dimensional (2D) panoramic ultrasound image 300 including a loss area 310 according to various embodiments. User input via the user input device 130 may trigger panoramic ultrasound image acquisition by the ultrasound probe 104. The ultrasound image acquisition may be a 2D or 3D image acquisition, such as acquiring a 2D panoramic ultrasound image, a 2D biplane image, or a 3D panoramic ultrasound image by the panoramic image acquisition processor 140. Additionally and / or alternatively, the panoramic ultrasound image acquisition may be a volumetric acquisition. Additionally and / or alternatively, the detected and located target structure 312 may be stored at the archive 138 and / or any suitable computer-readable medium. Additional ultrasound image acquisitions may be acquired automatically and / or in response to user input. Additional ultrasound image acquisitions may be 2D images, 3D images, or volumetric acquisitions. For example, if no user input is received within a predetermined time period, the panoramic image acquisition processor 140 may automatically initiate additional ultrasound image acquisitions by the ultrasound probe 104. As another example, the panoramic image acquisition processor 140 may be configured to initiate additional ultrasound image acquisition by the ultrasound probe 104 in response to user input.

[0034] Refer again Figure 1 , the panoramic image acquisition processor 140 may be configured to collect ultrasound image data as the ultrasound probe 104 slides over the region of interest, anatomical structure, and / or tissue. As the ultrasound probe 104 slides over the anatomical structure and / or tissue, the panoramic image acquisition processor 140 collects ultrasound images and stitches the ultrasound images together to generate a panoramic ultrasound image that includes the region of interest, anatomical structure, and / or tissue. For example, the panoramic image acquisition processor 140 may generate a captured panoramic ultrasound image 300. When the ultrasound probe 104 slides over the region of interest, the stitched ultrasound image may include a loss area 310 of a portion of the region of interest where the ultrasound probe cannot collect ultrasound image data due to the natural curvature of the human anatomy or other tissue.

[0035] The acquired panoramic ultrasound image 300 with the loss area 310 may be provided by the panoramic image acquisition processor 140 to the panoramic image generator processor 150. Additionally and / or alternatively, the generated image may be stored at the archive 138 and / or any suitable computer-readable medium, and the panoramic image generator processor 150 may obtain the panoramic ultrasound image from the archive 138 and / or any suitable computer-readable medium. In some examples, the panoramic image acquisition processor 140 may also be used to generate a reference panoramic ultrasound image to be stored in the archive 138, the training database 220, and / or any suitable computer-readable medium.

[0036] The signal processor 132 may include a panoramic image generator processor 150 including suitable logic, circuitry, interfaces, and / or code operable to obtain an acquired panoramic ultrasound image 300 from the panoramic image acquisition processor 140 and / or from the archive 138, the training database 220, and / or any suitable computer readable medium. For example, the panoramic image generator processor 150 may be configured to receive an acquired panoramic ultrasound image 300 having a loss area 310 from the panoramic image acquisition processor 140, or retrieve an acquired panoramic ultrasound image 300 having a loss area 310 from the archive 138 and / or the training database 220, and / or any suitable data storage medium.

[0037] Reference Figure 3 , Figure 3 4 is an exemplary two-dimensional (2D) reference panoramic ultrasound image 400. The reference panoramic ultrasound image 400 may have the same region of interest, anatomical structure, and / or tissue as the panoramic ultrasound image 300. The reference panoramic ultrasound image 400 may be a 2D image, a 2D biplane image, a 2D biplane image slice extracted from a volume, a rendered volume, and / or any suitable ultrasound image and / or volume rendering (i.e., a 2D projection of 3D / 4D volumetric image data). The reference panoramic ultrasound image 400 may be a complete reference ultrasound image without any loss area (e.g., image data 410 including a loss area 310 similar to the panoramic ultrasound image 300). The reference panoramic ultrasound image 400 may be obtained from the archive 138, the training database 220, and / or any suitable data storage medium. The archive 138, the training database 220, or other suitable data storage medium may include reference panoramic ultrasound images from various sources and may include various anatomical structures, tissues, and / or regions of interest of structures and / or tissues.

[0038] Reference Figure 1, the panoramic image generator processor 150 may include suitable logic components, circuits, interfaces, and / or code that may be configured to obtain the panoramic ultrasound image 300 as an input and learn the probability distribution to generate a sample panoramic ultrasound image. For example, the panoramic image generator processor 150 may obtain the panoramic ultrasound image 300 from the panoramic image acquisition processor 140, the archive 138, and / or any other suitable computer-readable medium. The panoramic image generator processor 150 generates sample panoramic ultrasound images that appear to be real (although some sample images are generated and some are real) and are provided to the panoramic image discriminator processor 160. Additionally or alternatively, the sample panoramic ultrasound images may be stored in the archive 138, the training database 220, and / or any other suitable computer-readable medium, and the panoramic image discriminator processor 160 may retrieve the sample panoramic ultrasound images from the archive 138, the training database 220, and / or any other suitable computer-readable medium.

[0039] Still refer to Figure 1 , the panoramic image discriminator processor 160 may obtain as input a reference panoramic ultrasound image 400 from the panoramic image generator processor 150, the training database 220, the archive 138, and / or any other suitable computer-readable medium. The panoramic image discriminator processor 160 may also be configured to obtain sample panoramic ultrasound images from the panoramic image generator processor 150 and use the sample panoramic ultrasound images to learn to distinguish between real images and generated images. The panoramic image generator processor 150 may also be operable to obtain reference panoramic ultrasound images from the training database 220, from the archive 138, and / or from any suitable data storage medium.

[0040] Figure 4 is a graphical representation 500 of an exemplary deep learning technique for improved panoramic ultrasound images according to various embodiments. Figure 4 , with a loss area 510 (e.g., Figure 3 The panoramic ultrasound image 150 of FIG. 300 is provided as an input to the panoramic image generator processor 150. The panoramic image generator processor 150 generates a sample panoramic ultrasound image 152, which is provided as an input to the panoramic image discriminator processor 160.

[0041] In some examples, the deep learning technique is a generative adversarial network (GAN) model, and the GAN model is used to train the panoramic image generator processor 150 and the panoramic image discriminator processor 160. For example, the panoramic image generator processor 150 may be configured to generate a sample panoramic ultrasound image 152 by mapping a panoramic ultrasound image with a loss area 510 to a latent space to learn a probability distribution, which the panoramic image generator processor 150 may use to generate a sample panoramic ultrasound image 152 that looks real. The panoramic image generator 150 may be configured to learn a probability distribution that approximates the true distribution p r The distribution of (x)p θ (x), and generate equal to the real sample p r The sample p of the probability density function of (x) G (x). For example, the panoramic image generator 150 may directly learn the differential function p θ (x), so that p θ (x)>0 and ∫ x p θ (x)d(x)=1, and is optimized by maximum likelihood. Additionally or alternatively, as a non-limiting example, the panoramic image generator may learn p θ The differential transformation function q of (x) θ (z), and is optimized by maximum likelihood when z is a distribution such as a uniform distribution or a Gaussian distribution. The panoramic image generator processor 150 may provide a sample panoramic ultrasound image 152 to the panoramic image discriminator processor 160 .

[0042] The panoramic image discriminator processor 160 may receive as input the sample panoramic ultrasound image 152 and the reference panoramic ultrasound image 520 from the panoramic image generator processor 150 and learn to distinguish between real images and generated images. The panoramic image discriminator processor 160 may classify the received image as a real image or a generated image by outputting a value D(x) 170. In some examples, the value is a binary value (e.g., "0" or "1"). The panoramic image discriminator processor 160 may identify a real image or a generated image from a real data distribution p r (x), where D indicates the estimated probability of a data point x i ∈R n In the case of binary classification, if the estimated probability D(x i ):->R n [0, 1] is the positive class p i And 1-D(x i ):->R[0,1] is the negative class qi, then the cross entropy distribution between pi and qi is For a given point x i and the corresponding label y i, data distribution x i Can come from real data x i ~p r (x) or generator data x i ~p g (z). The panoramic image generator processor 150 and the panoramic image discriminator processor 160 may have an adversarial relationship, where the panoramic image generator processor 150 produces a fake generated image and the panoramic image discriminator processor 160 learns to distinguish between real images and generated images. Considering that half of the data from the panoramic image generator processor 150 and the panoramic image discriminator processor 160 is real and half is generated, the panoramic image generator processor 150 and the panoramic image discriminator processor 160 compete with each other to minimize the loss function.

[0043] D(x i ) may be provided as feedback to the panoramic image discriminator processor 160 and the panoramic image generator processor 150. Additionally and / or alternatively, one or more cost functions and / or loss functions may be used to provide feedback to the panoramic image generator processor 150 and the panoramic image discriminator processor 160. The loss function may be, for example:

[0044]

[0045] in is the coefficient to overcome the vanishing gradient effect.

[0046] By competing and receiving feedback, the panoramic image generator processor 150 can generate images similar to real ultrasound images, and the panoramic image discriminator processor 160 becomes better at distinguishing between real ultrasound images and generated ultrasound images. The goal is to enable the panoramic image generator processor 150 to generate images that are similar to real ultrasound images, at which point the panoramic image discriminator processor 160 may also be unable to distinguish between real ultrasound images and generated ultrasound images, and the training can be considered complete.

[0047] Once the training of the panoramic image generator processor 150 and the panoramic image discriminator processor 160 using the GAN model is completed, the panoramic image generator processor 150 may generate an enhanced panoramic ultrasound image 530. For example, the panoramic image generator processor 150 may obtain an acquired panoramic ultrasound image from the panoramic image acquisition processor 140 and generate an enhanced panoramic ultrasound image 530.

[0048] Refer again Figure 1The display system 134 may be any device capable of conveying visual information to a user. For example, the display system 134 may include a liquid crystal display, a light emitting diode display, and / or any suitable one or more displays. The display system 134 may be operable to present a 2D ultrasound image, a 2D panoramic ultrasound image 300, 400, a bi-plane ultrasound image, a bi-plane ultrasound slice extracted from a 3D / 4D volume, a rendered 3D / 4D volume, a selectable target structure, and / or any suitable information.

[0049] The archive 138 may be one or more computer-readable memories, such as a picture archiving and communication system (PACS), a server, a hard disk, a floppy disk, a CD, a CD-ROM, a DVD, a compact memory, a flash memory, a random access memory, a read-only memory, an electrically erasable and programmable read-only memory, and / or any suitable memory, that are integrated with the ultrasound system 100 and / or communicatively coupled (e.g., via a network) to the ultrasound system 100. The archive 138 may include, for example, a database, a library, a collection of information, or other memory that is accessed by and / or in conjunction with the signal processor 132. For example, the archive 138 may be capable of storing data temporarily or permanently. The archive 138 may be capable of storing medical image data, data generated by the signal processor 132, and / or instructions readable by the signal processor 132, etc. In various embodiments, the archive 138 stores, for example, 2D ultrasound images, 2D panoramic ultrasound images 300, 400, bi-plane ultrasound images, bi-plane ultrasound slices extracted from a 3D / 4D volume, rendered 3D / 4D volumes, instructions for acquiring ultrasound image data, instructions for generating panoramic ultrasound images, instructions for generating sample panoramic ultrasound images, instructions for classifying an image as a generated image or a real image, instructions for providing feedback based on the classification of the image, instructions for determining that a target function has been achieved, and instructions for generating an enhanced panoramic ultrasound image.

[0050] The components of the ultrasound system 100 may be implemented in software, hardware, firmware, etc. The various components of the ultrasound system 100 may be communicatively connected. The components of the ultrasound system 100 may be implemented separately and / or integrated in various forms. For example, the display system 134 and the user input device 130 may be integrated into a touch screen display.

[0051] Still refer to Figure 1, the training system 200 may include a training engine 210 and a training database 220. The training engine 210 may include suitable logic components, circuits, interfaces, and / or code that are operable to train neurons of a deep neural network (e.g., an artificial intelligence model) inferred (i.e., deployed) by the panoramic image acquisition processor 140, the panoramic image generator processor 150, and / or the panoramic image generator processor 160. For example, the artificial intelligence model inferred by the panoramic image generator processor 150 and / or the panoramic image discriminator processor 160 may be trained to automatically acquire ultrasound images and / or volumes using the database 220 of classified ultrasound images and / or volumes of anatomical structures. As another example, the artificial intelligence model inferred by the panoramic image acquisition processor may be trained to automatically identify a target structure, surrounding structures, target structure shape, major axis / minor axis of a target structure, etc. depicted in an ultrasound volume using the database 220 of classified ultrasound volumes of possible target structures.

[0052] In various embodiments, the database 220 of training images may be a picture archiving and communication system (PACS) or any suitable data storage medium. In certain embodiments, the training engine 210 and / or the training image database 220 may be a remote system communicatively coupled to the ultrasound system 100 via a wired or wireless connection, such as Figure 1 Additionally and / or alternatively, components or all of the training system 200 may be integrated with the ultrasound system 100 in various forms. In some examples, the training image database may include reference panoramic ultrasound images of anatomical structures and / or tissues. In some examples, the reference panoramic ultrasound images may be generated by the panoramic image acquisition processor 140 and provided to the training image database 220.

[0053] Figure 5 600, according to various embodiments, illustrates exemplary steps 602-616 that may be used to train deep learning techniques to enhance panoramic ultrasound images. Certain embodiments may omit one or more steps, and / or perform the steps in a different order than listed, and / or combine certain steps discussed below. For example, some steps may not be performed in certain embodiments. As another example, certain steps may be performed in a different time order than listed below, including simultaneously.

[0054] At step 602, the signal processor 132 of the ultrasound system 100 may be configured to initiate training to enhance the panoramic ultrasound image. For example, at step 604, the panoramic image acquisition processor 140 may be configured to collect ultrasound image data when the ultrasound probe 104 slides over the region of interest of the anatomical structure and / or tissue. When the ultrasound probe 104 slides over the region of interest, the panoramic image acquisition processor 140 collects ultrasound image data to generate an ultrasound image.

[0055] At step 606, the signal processor 132 may be configured to generate an acquired panoramic ultrasound image 300. For example, as the ultrasound probe 104 slides over the region of interest, anatomical structure, and / or tissue, the panoramic image acquisition processor 140 stitches the ultrasound images together to generate the acquired panoramic ultrasound image 300. The acquired panoramic ultrasound image 300 may include a loss area 310 for a portion of the region of interest, anatomical structure, and / or tissue where the ultrasound probe is unable to collect ultrasound image data due to the curvature of the human anatomy or other tissue. The panoramic image acquisition processor 140 may generate the acquired panoramic ultrasound image 300 and provide the acquired panoramic ultrasound image 300 to the panoramic image generator processor 150.

[0056] At step 608, the signal processor 132 may be configured to generate a sample panoramic ultrasound image using the acquired panoramic ultrasound image 300. For example, the panoramic ultrasound image generator processor 150 may be configured to map the sample panoramic ultrasound image to a latent space and learn a probability distribution that the panoramic image generator processor 150 may use to generate the sample panoramic ultrasound image. The panoramic image generator processor 150 may provide the sample panoramic ultrasound image to the panoramic image discriminator processor 160.

[0057] At step 610, the signal processor 132 may be configured to classify the sample panoramic ultrasound image. For example, the panoramic image discriminator processor 160 may receive the sample panoramic ultrasound image and the reference panoramic ultrasound image (e.g., Figure 4 The panoramic image discriminator processor 160 may classify the received image as a real image or a generated image.

[0058] At step 612, the signal processor 132 may be configured to use the results of the classification in order to provide feedback to the panoramic image generator processor 150 and the panoramic image discriminator processor 160. For example, the panoramic image discriminator processor 160 may classify the received image as a real image or a generated image by outputting a value D(x). In some examples, the value is a binary value (e.g., "0" or "1"). D(x) may be provided to the panoramic image discriminator processor 160 and the panoramic image generator processor 150 as feedback. Additionally and / or alternatively, one or more cost functions and / or loss functions may be used to provide feedback to the panoramic image generator processor 150 and the panoramic image discriminator processor 160.

[0059] At step 614, the signal processor 132 may be configured to repeat steps 608-612 until the objective function is achieved. For example, the panoramic image generator processor 150 and the panoramic image discriminator processor 160 may have an adversarial relationship, wherein the panoramic image generator processor 150 generates a false / generated image, and the panoramic image discriminator processor 160 learns to distinguish between a real panoramic ultrasound image and a generated panoramic ultrasound image. Through competition and receiving feedback, the panoramic image generator processor 150 may generate a panoramic ultrasound image similar to a real panoramic ultrasound image, and the panoramic image discriminator processor 160 becomes better at distinguishing between a real panoramic ultrasound image and a generated panoramic ultrasound image. The goal is to make the generated panoramic ultrasound image from the panoramic image generator processor 150 similar to a real ultrasound image, and / or to make the panoramic image discriminator processor 160 unable to distinguish between a real ultrasound image and a generated ultrasound image. Once the objective function is achieved, the training is completed at step 616.

[0060] Figure 6 is a flowchart 700 illustrating exemplary steps that may be used to enhance panoramic ultrasound images using deep learning, according to various embodiments. Figure 6 700 is a flowchart illustrating exemplary steps 702-706 that may be used to enhance panoramic ultrasound images using deep learning, according to various embodiments. Certain embodiments may omit one or more steps, and / or perform the steps in a different order than listed, and / or combine certain steps discussed below. For example, some steps may not be performed in certain embodiments. For another example, certain steps may be performed in a different time order than listed below, including simultaneously.

[0061] At step 702, the signal processor 132 of the ultrasound system 100 may be configured to collect ultrasound image data as the ultrasound probe 104 slides over the region of interest of the anatomical structure and / or tissue. As the ultrasound probe 104 slides over the region of interest, the panoramic image acquisition processor 140 collects ultrasound image data and stitches the ultrasound images together to produce the acquired panoramic ultrasound image 500.

[0062] At step 704, the signal processor 132 may be configured to generate an acquired panoramic ultrasound image 500. For example, when the ultrasound probe slides over the region of interest, anatomical structure, and / or tissue, the acquired panoramic ultrasound image may include a lost area 310 of a portion of the region of interest where the ultrasound probe 104 is unable to collect ultrasound image data due to the curvature of the human anatomy or other tissue. The panoramic image acquisition processor 140 may generate the acquired panoramic ultrasound image 500 and provide the acquired panoramic ultrasound image 500 to the panoramic image generator processor 150.

[0063] At step 706, the signal processor 132 may be configured to generate an enhanced panoramic ultrasound image. For example, the image may be generated by using, for example, Figure 6 The panoramic image generator processor 150 trained by the method of the invention may receive the sample panoramic ultrasound image 500 from the panoramic image acquisition processor 140 and may generate an enhanced panoramic ultrasound image. The enhanced panoramic ultrasound image generated by the panoramic image acquisition processor 140 may be similar to the acquired panoramic ultrasound image, except that the enhanced panoramic ultrasound image is complete and does not include the loss area.

[0064] Aspects of the present disclosure provide methods 600, 700 and systems 100 for enhancing panoramic ultrasound images. According to various embodiments, the methods 600, 700 may include: acquiring 604, 702 ultrasound image data of an anatomical structure by an ultrasound probe 104 of the ultrasound system 100; generating 606, 704 an acquired panoramic ultrasound image 500 of the anatomical structure based on the ultrasound image data by at least one processor 132, 140 of the ultrasound system 100, the acquired panoramic ultrasound image 500 including the loss area 310, 410; processing 608, 706 the acquired panoramic ultrasound image 500 by at least one processor 132, 150 of the ultrasound system 100 using one or more reference panoramic ultrasound images 400 without the loss area 310, 410, 520; and outputting an enhanced panoramic ultrasound image without the loss area 310, 410, 550 by at least one processor 132, 150 of the ultrasound system 100 for presentation on the display system 134. The method 600, 700 may include 2D or 3D ultrasound image data. The method 600, 700 may use a generative adversarial network (GAN) model 152, 170, 602 to process the acquired panoramic ultrasound image 300. The GAN model 152, 170, 602 may include: generating 608 one or more sample panoramic ultrasound images 152, 608 by at least one processor 132, 150 of the ultrasound system 100; obtaining one or more reference panoramic ultrasound images 400 without loss areas 410, 520; classifying 610 the one or more sample panoramic ultrasound images 152 and the one or more reference panoramic ultrasound images 520 by at least one processor 132, 160 of the ultrasound system 100; and providing feedback 170, 612 based on the classification 170, 610 of the one or more sample panoramic ultrasound images 152 and the one or more reference panoramic ultrasound images 400, 520. The feedback 170, 612 may include an error rate 170. The feedback 170 , 612 including the error rate may be a cost function, a loss function, or a probability distribution 170 .

[0065] The method 600 , 700 may include learning the loss area 310 , 410 of the one or more reference panoramic ultrasound images 400 using deep learning, wherein the loss area 310 , 410 is repeatedly learned using the one or more reference panoramic ultrasound images until a target function is reached 614 .

[0066] Various embodiments provide an ultrasound system 100, the ultrasound system comprising: an ultrasound probe 104 configured to acquire 604, 702 ultrasound image data of an anatomical structure; and at least one processor 132, 140, the at least one processor configured to: generate 606, 704 an acquired panoramic ultrasound image 300 of an anatomical structure based on the ultrasound image data, the acquired panoramic ultrasound image 300 including a loss area 310, 410; process 608, 706 the acquired panoramic ultrasound image 300 using one or more reference panoramic ultrasound images 400 without the loss area 310, 410; and output an enhanced panoramic ultrasound image without the loss area 310, 410, 530 for presentation on a display system 134. The ultrasound image data of the ultrasound system 100 may be 2D or 3D. The at least one processor 132, 150 may also be configured to process the acquired panoramic ultrasound image 300 using a generative adversarial network (GAN) model 152, 170, 602. At least one processor 132, 150 may also be configured to: generate 608 one or more sample panoramic ultrasound images 152, 608; obtain one or more reference panoramic ultrasound images 400 without loss area 410; classify 610 one or more sample panoramic ultrasound images 152, 608 and one or more reference panoramic ultrasound images 400, 520; and provide feedback 170, 612 based on the classification 170, 610 of one or more sample panoramic ultrasound images 152 and one or more reference panoramic ultrasound images 520.

[0067] The feedback 170, 612 of the ultrasound system 100 may be a cost function, a loss function, or a probability distribution 170. The feedback 170, 612 may be used to generate additional sample panoramic images 152, 608. The at least one processor 132, 160 may also be configured to learn the loss area 310, 410 of the one or more reference panoramic ultrasound images 400, 520 using deep learning, wherein the learning of the loss area 310, 410 is repeated by processing the one or more reference panoramic ultrasound images until the objective function is reached 614.

[0068] Various embodiments provide an ultrasound system 100 for enhancing panoramic ultrasound images, the ultrasound system comprising: an ultrasound probe 104 configured to acquire 604, 702 ultrasound image data of an anatomical structure; and at least one processor 132, 140 configured to: generate 606, 704 an acquired panoramic ultrasound image 300 of an anatomical structure based on the ultrasound image data, the acquired panoramic ultrasound image 300 including a loss area 310; learn the loss area using deep learning by processing one or more reference panoramic ultrasound images 400 without the loss area 410; process the acquired panoramic ultrasound image 300 using the learned loss area 410, 520; and output an enhanced panoramic ultrasound image without the loss area 310, 530 for presentation on a display system 134. In some examples, the ultrasound image data is 2D or 3D. The deep learning of the ultrasound system can be performed using a generative adversarial network (GAN) model 152, 170, 602. The at least one processor 132, 150 of the ultrasound system 100 may also be configured to learn the loss area 310, 410 by: generating 608 one or more sample panoramic ultrasound images 152, 608; obtaining one or more reference panoramic ultrasound images 400 without the loss area 410; classifying 610 the one or more sample panoramic ultrasound images 152 and the one or more reference panoramic ultrasound images 400; and providing feedback 170, 612 based on the classification 170, 610 of the one or more sample panoramic ultrasound images 152 and the one or more reference panoramic ultrasound images 400, 520. The feedback 170, 612 of the ultrasound system 100 may be a cost function, a loss function, or a probability distribution 170. The feedback may be used to generate additional sample panoramic images 152, 608.

[0069] As used herein, the term "circuit" refers to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure hardware, is executed by hardware, and / or is otherwise associated with hardware. For example, as used herein, when executing one or more first codes, a specific processor and memory may include a first "circuit", and when executing one or more second codes, a specific processor and memory may include a second "circuit". As used herein, "and / or" represents any one or more of the items in the list connected by "and / or". For example, "x and / or y" represents any element in the three-element set {(x), (y), (x, y)}. As another example, "x, y, and / or z" represents any element in the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. As used herein, the term "exemplary" means used as a non-limiting example, instance, or illustration. As used herein, the terms "for example" and "such as" introduce a list of one or more non-limiting examples, instances, or illustrations. As used herein, a circuit is "operable to" and / or "configured to" perform a function whenever the circuit includes the necessary hardware and code (if necessary) to perform the function, regardless of whether the execution of the function is disabled or not enabled by some user-configurable setting.

[0070] Other embodiments may provide a computer-readable device and / or a non-transitory computer-readable medium, and / or a machine-readable device and / or a non-transitory machine-readable medium, on which are stored machine codes executable by a machine and / or a computer and / or a computer program having at least one code segment, so that the machine and / or the computer performs the steps for enhancing panoramic ultrasound images using deep learning as described herein.

[0071] Therefore, the present disclosure may be implemented in hardware, software, or a combination of hardware and software. The present disclosure may be implemented in a centralized manner in at least one computer system, or in a distributed manner, where different elements are distributed across several interconnected computer systems. Any type of computer system or other device suitable for executing the methods described herein is suitable.

[0072] The various embodiments may also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein and which, when loaded into a computer system, is capable of carrying out these methods. A computer program herein refers to any expression of a set of instructions in any language, code or notation, which is intended to cause a system with information processing capabilities to perform specific functions directly or after either or both of the following: a) conversion into another language, code or notation; b) reproduction in a different material form.

[0073] Although the present disclosure has been described with reference to certain embodiments, it will be appreciated by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the present disclosure. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of the present disclosure without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the specific embodiments disclosed, but the present disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. A method (600, 700) for enhancing a panoramic ultrasound image, the method comprising: Acquiring (604, 702) ultrasound image data of an anatomical structure by an ultrasound probe (104) of an ultrasound system (100); generating (606, 704), by at least one processor (132, 140) of the ultrasound system (100), an acquired panoramic ultrasound image (300) of the anatomical structure based on the ultrasound image data, the acquired panoramic ultrasound image (300) including a loss area (310); processing (608, 706) the acquired panoramic ultrasound image by the at least one processor (132, 150) of the ultrasound system (100) using one or more reference panoramic ultrasound images (400) without the loss area (310, 410); as well as An enhanced panoramic ultrasound image without the lost area (310, 410, 530) is output by the at least one processor (132, 150) of the ultrasound system (100) for presentation on a display system (134).

2. The method (600, 700) of claim 1, wherein processing the acquired panoramic ultrasound image is performed using a generative adversarial network (GAN) model (152, 170, 500).

3. The method (600, 700) of claim 2, wherein the GAN model comprises: generating (608) one or more sample panoramic ultrasound images (152, 608) by at least one processor (132, 150) of the ultrasound system (100); obtaining the one or more reference panoramic ultrasound images (400) without the loss area (410, 520); classifying (610), by the at least one processor (132, 160) of the ultrasound system (100), the one or more sample panoramic ultrasound images (152, 608) and the one or more reference panoramic ultrasound images (520); and Feedback (170, 612) is provided based on the classification (610) of the one or more sample panoramic ultrasound images (152, 608) and the one or more reference panoramic ultrasound images (400, 520).

4. The method (600, 700) of claim 3, wherein the feedback (170, 612) comprises an error rate.

5. The method (600, 700) of claim 4, wherein the feedback (170, 612) including the error rate is a cost function, a loss function or a probability distribution (170).

6. According to the method (600, 700) of claim 1, the method also includes using deep learning to learn the loss area (310, 410) of the one or more reference panoramic ultrasound images (400), wherein the learning of the loss area (310, 410) is repeated using the one or more reference panoramic ultrasound images until the objective function is reached (614).

7. An ultrasound system (100), comprising: an ultrasound probe (104) configured to acquire (604, 702) ultrasound image data of an anatomical structure; and At least one processor (132), the at least one processor configured to: generating (606, 704) an acquired panoramic ultrasound image (300) of the anatomical structure based on the ultrasound image data, the acquired panoramic ultrasound image (300) including a loss area (310); processing (608, 706) the acquired panoramic ultrasound image (300) using one or more reference panoramic ultrasound images (400) without the loss area (410); and An enhanced panoramic ultrasound image without the loss area (310, 410, 530) is output for presentation on a display system (134).

8. The ultrasound system (100) of claim 7, wherein the at least one processor (132, 150) is further configured to process the acquired panoramic ultrasound image using a generative adversarial network (GAN) model (152, 170, 500).

9. The ultrasound system (100) of claim 7, wherein the at least one processor (132, 150) is further configured to: generating (608) one or more sample panoramic ultrasound images (152, 608); obtaining the one or more reference panoramic ultrasound images (400) without the loss area (410); classifying (610) the one or more sample panoramic ultrasound images (152, 608) and the one or more reference panoramic ultrasound images (400, 520); and Feedback (170.612) is provided based on the classification (610) of the one or more sample panoramic ultrasound images (152, 608) and the one or more reference panoramic ultrasound images (520).

10. The ultrasound system (100) of claim 9, wherein the feedback (170, 612) is a cost function, a loss function, or a probability distribution (170).

11. The ultrasound system (100) of claim 9, wherein the feedback is used to generate additional sample panoramic images (152, 608).

12. The ultrasound system (100) of claim 7, wherein the at least one processor (132, 160) is further configured to learn the loss area (310, 410) of the one or more reference panoramic ultrasound images (400, 520) using deep learning, wherein the learning of the loss area (310, 410) is repeated by processing the one or more reference panoramic ultrasound images until a target function is reached (614).

13. An ultrasound system (100) for enhancing panoramic ultrasound images, the ultrasound system comprising: An ultrasound probe (104), the ultrasound probe being configured to: acquiring (604, 702) ultrasound image data of an anatomical structure; and at least one processor (132, 140), the at least one processor being configured to: generating (606, 704) an acquired panoramic ultrasound image of the anatomical structure based on the ultrasound image data, the acquired panoramic ultrasound image (300) including a loss area (310); learning the loss area using deep learning by processing one or more reference panoramic ultrasound images (400) without the loss area (410); Processing the acquired panoramic ultrasound image using the learned loss area (410, 520); and An enhanced panoramic ultrasound image without the loss area (310, 530) is output for presentation on a display system (134).

14. The ultrasound system (100) of claim 13, wherein the deep learning is performed using a generative adversarial network (GAN) model (152, 500).

15. The ultrasound system (100) of claim 13, wherein the at least one processor is further configured to learn the loss area by: generating (608) one or more sample panoramic ultrasound images (152, 608); obtaining the one or more reference panoramic ultrasound images (400) without the loss area (410); classifying (610) the one or more sample panoramic ultrasound images (152, 608) and the one or more reference panoramic ultrasound images (400, 520); and Feedback (170, 612) is provided based on the classification of the one or more sample panoramic ultrasound images (152, 608) and the one or more reference panoramic ultrasound images (520).

Images