Guided ultrasound imaging

Abstract

The present disclosure describes imaging systems configured to generate volumetric images of a target feature based on anatomical landmarks identified during an ultrasound scan and according to a user-selected view. The system can include an ultrasound transducer configured to acquire echo signals in response to ultrasound pulses emitted toward a target region. A processor coupled with the transducer can present illustrative volumetric images of the target feature for user selection, each image corresponding to a particular view. The processor can then identify anatomical landmarks corresponding to the target feature embodied within 2D image frames and provide instructions for steering the transducer to a target location to generate 2D image frames specific to the selected view based on the identified landmarks and the user-selected view. Echo signals are then acquired at the target region and used to generate actual volumetric images of the target feature corresponding to the user-selected view.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to ultrasound systems and methods for identifying anatomical features via ultrasound imaging and guiding a user to capture a desired view of a target feature. The DETAILED DESCRIPTION can utilize at least one neural network configured to generate customized user instructions based on anatomical features identified in a current ultrasound image and associated displays. Embodiments further include ultrasound image acquisition components configured to automatically switch between 2D imaging mode and volumetric imaging mode. BACKGROUND

[0002] Existing ultrasound systems can produce 3D images by acquiring a panorama of 2D images and stitching them together. This particular imaging method is particularly common in prenatal applications designed to produce 3D images of babies in the womb. In particular, it is often desirable to capture detailed images of a baby’s face to provide a first glimpse of an unborn baby. Unfortunately, acquiring quality images of the face can be highly dependent on the baby’s position within the womb, and ultrasound operators often lack training on the ways to manipulate the ultrasound transducer needed to overcome variations in the baby’s position. Thus, new techniques configured to produce quality images of various anatomical features, such as a face of an unborn baby, regardless of the orientation of such features. SUMMARY

[0003] The present disclosure describes systems and methods for capturing ultrasound images of various anatomical objects according to a particular view selected by a user. Although the examples herein specifically address prenatal imaging of a fetus to collect images of the face of an unborn infant, those skilled in the art will appreciate that the disclosed systems and methods are described with respect to fetal imaging for illustrative purposes only, and that anatomical imaging can be performed according to the present disclosure on various anatomical features including, but not limited to, for example, the heart and lungs. In some embodiments, a system can be configured to improve the accuracy, efficiency, and automation of prenatal ultrasound imaging by guiding a user to collect images that target anatomical features, automatically selecting regions of interest (ROIs) within the images of the features, capturing volumetric (e.g., 3D) images of the features, and modifying the volumetric images according to user preferences. The system can include an ultrasound transducer configured to collect echo signals in response to ultrasound pulses emitted toward a target region, which can include the abdomen of a patient. One or more processors can be coupled with the ultrasound transducer, each processor uniquely configured to perform one or more functions based on ultrasound data collected by the transducer. For example, a data processor can be configured to implement one or more neural networks configured to identify certain anatomical features and guide an operator to manipulate the transducer in a manner needed to collect images of the target features. Additionally or alternatively, the data processor can be configured to perform image segmentation or another boundary detection technique to identify certain anatomical features. After defining ROIs within the images, a control circuit can be configured to automatically switch the transducer into a volumetric imaging mode to sweep through the ROIs. The collected volumetric images can be modified, for example, by a neural network or image rendering processor configured to apply certain image modifications for improved clarity, quality, and / or artistic purposes. Although particular embodiments are described herein with respect to generating 3D images, the present disclosure is not limited to 3D imaging. For example, embodiments can also relate to additional forms of volumetric imaging, such as 4D and / or spatio-temporal image correlation (STIC) imaging.

[0004] According to some examples of the present disclosure, an ultrasound imaging system can include an ultrasound transducer configured to acquire echo signals responsive to ultrasound pulses transmitted toward a target region. The system can also include one or more processors in communication with the ultrasound transducer and configured to: present one or more illustrative volume images of a target feature to a user, each illustrative volume image corresponding to a particular view of the target image; receive a user selection of one of the illustrative volume images; generate two-dimensional (2D) image frames from the acquired echo signals of the target region; identify one or more anatomical landmarks in the generated 2D image frames that correspond to the target feature; provide instructions for steering the ultrasound transducer to a target position for generating at least one 2D image frame specific to the particular view of the user-selected volume image based on the anatomical landmarks and the particular view of the user-selected volume image; cause the ultrasound transducer to acquire additional echo signals at the target position; and generate an actual volume image of the target feature and corresponding to the particular view of the user-selected volume image using the acquired additional echo signals.

[0005] In some examples, the one or more processors are configured to identify the one or more anatomical landmarks via image segmentation. In some examples, the one or more processors are configured to identify the one or more anatomical landmarks via an implementation of a neural network trained to identify the anatomical landmarks. In some examples, the one or more processors are further configured to apply artificial lighting to the actual volume image according to the particular view. In some examples, the artificial lighting is applied by an artificial neural network. Examples can include one or more artificial neural networks, such as two, three, or more communicatively coupled neural networks. In some embodiments, the artificial neural network can also be configured to apply image contrast adjustments to the actual volume image according to the particular view. In some examples, the target feature can include a face of an unborn infant.

[0006] In some examples, the one or more processors are configured to generate the instructions for manipulating the ultrasound transducer by inputting the 2D image frame to an artificial neural network trained to compare the 2D image frame to stored image frames embodying the target feature. In some examples, the artificial neural network is configured to generate new instructions for manipulating the ultrasound transducer in the event of repositioning of the ultrasound transducer. In some examples, the one or more processors are further configured to define a region of interest within the 2D image frame specific to the particular view of the volume image selected by the user. In some examples, the ultrasound imaging system further comprises a controller configured to switch the ultrasound transducer from a 2D imaging mode to a volume imaging mode. The controller can be configured to automatically switch the ultrasound transducer from the 2D imaging mode to the volume imaging mode in the event of receiving an indication from the one or more processors that the region of interest has been defined. Embodiments can also include a user interface communicatively coupled with the one or more processors and configured to display the instructions for manipulating the ultrasound transducer to a target position. In some embodiments, the one or more processors can be further configured to cause an indicator of the target feature to be displayed on the user interface.

[0007] According to some embodiments, a method of ultrasound imaging can involve acquiring echo signals in response to ultrasound pulses emitted toward a target region. The method can also involve presenting one or more illustrative volume images of a target feature to a user, each illustrative volume image corresponding to a particular view of the target image; receiving a user selection of one of the illustrative volume images; generating a two-dimensional (2D) image frame from the acquired echo signals of the target region; identifying one or more anatomical landmarks in the generated 2D image frame corresponding to the target feature; providing instructions for manipulating the ultrasound transducer to a target position for generating at least one 2D image frame specific to the particular view of the user-selected volume image based on the anatomical landmarks and the particular view of the user-selected volume image; causing the ultrasound transducer to acquire additional echo signals at the target position; and generating an actual volume image of the target feature and corresponding to the particular view of the user-selected volume image using the acquired additional echo signals.

[0008] In some examples, the method further involves applying artificial lighting, image contrast adjustment, or both to the actual volumetric image. In some examples, the target feature includes a face of an unborn infant. In some examples, identifying the one or more anatomical landmarks involves image segmentation or implementation of at least one neural network trained to identify the anatomical landmarks. In some examples, the method further involves displaying instructions for manipulating the ultrasound transducer. Embodiments can also involve defining a region of interest within the 2D image frame specific to the particular view of the volumetric image selected by the user. In some examples, the method further involves identifying additional anatomical landmarks of the target feature with manipulation of the ultrasound transducer; and generating additional instructions for manipulating the ultrasound transducer based on the additional anatomical landmarks identified with manipulation of the ultrasound transducer. Example methods can also involve switching the ultrasound transducer from the 2D imaging mode to a volumetric imaging mode upon receiving an indication that a region of interest has been identified.

[0009] Any of the methods described herein, or steps thereof, can be embodied in a non-transitory computer-readable medium comprising executable instructions that, when executed, can cause a processor of a medical imaging system to perform the methods or steps embodied herein. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a block diagram of an ultrasound system in accordance with the principles of the present disclosure.

[0011] Figure 2 is a display of a target feature viewing option presented on a user interface in accordance with the principles of the present disclosure.

[0012] Figure 3 is a display of user instructions for ultrasound probe manipulation overlaid on a live 2D image generated in accordance with the principles of the present disclosure.

[0013] Figure 4 is a display of another user instruction for ultrasound probe manipulation overlaid on a live 2D image generated in accordance with the principles of the present disclosure.

[0014] Figure 5 is a display of another user instruction for ultrasound probe manipulation overlaid on a live 2D image generated in accordance with the principles of the present disclosure.

[0015] Figure 6 is a display showing automatic positioning of a region of interest on a live 2D image in accordance with the principles of the present disclosure.

[0016] Figure 7 is a display of an additional target feature viewing option presented on a user interface in accordance with the principles of the present disclosure.

[0017] Figure 8 is a flowchart of a method of ultrasound imaging performed in accordance with the principles of the present disclosure. DETAILED DESCRIPTION

[0018] The following description of certain embodiments is merely exemplary in nature and is in no way intended to limit the application or its applications or uses. In the following detailed description of embodiments of the present system and method, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the described system and method can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the presently disclosed system and method, and it is to be understood that other embodiments can be utilized and that structural and logical changes can be made without departing from the spirit and scope of the present system. Furthermore, specific terminology used herein is to be construed in a manner consistent with the principles of the present system, and in no way limiting, such that the specific features of the present system are to be understood as being among others. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present system is defined only by the appended claims.

[0019] The ultrasound system according to the present disclosure can utilize one or more artificial neural networks implemented by a computer processor, module, or circuit. Example networks include convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), autoencoder neural networks, and the like, configured to identify one or more anatomical features (e.g., a head, foot, hand, or leg) present within a 2D ultrasound image and guide a user to manipulate an ultrasound transducer in a manner required to capture an image of a particular targeted anatomical feature (e.g., the face of an unborn baby). The artificial neural network(s) can be trained using any of a variety of currently known or later developed machine learning techniques to obtain a neural network (e.g., a machine-trained algorithm or hardware-based node system) configured to analyze input data in the form of ultrasound image frames and determine imaging adjustments required to acquire a particular view of at least one anatomical feature. Neural networks can provide advantages over conventional computer programming algorithms in that they can be generalized and trained to recognize features of a data set by analyzing samples of the data set rather than by relying on specialized computer code. By presenting the neural network training algorithm with appropriate input and output data, the one or more neural networks of the ultrasound system according to the present disclosure can be trained to recognize a plurality of anatomical features, guide a user to obtain an image of a target feature based in part on the recognized anatomical features, refine a ROI encompassing the target feature, and / or obtain a 3D image of the target feature.

[0020] An ultrasound system according to the principles of the present disclosure can include or be operatively coupled to an ultrasound transducer configured to emit ultrasound pulses toward a medium (e.g., a human body or a particular portion thereof) and generate echo signals in response to the ultrasound pulses. The ultrasound system can include a beamformer configured to perform transmit and / or receive beamforming, and a display configured to display ultrasound images generated by the ultrasound imaging system and notifications overlaid on or adjacent to the images. The ultrasound imaging system can include one or more processors and at least one neural network, which can be implemented in hardware and / or software components. The neural network(s) can be machine trained to identify anatomical features present within 2D images, guide a user to obtain images of a target feature, identify regions of interest within images of the target feature, and / or modify 3D images of the target feature.

[0021] The neural network(s) according to some example implementations of the present disclosure can be hardware-based (e.g., neurons represented by physical components) or software-based (e.g., neurons and pathways implemented in a software application), and can be trained using various topologies and learning algorithms to produce a desired output. For example, a software-based neural network can be implemented using a processor (e.g., a single- or multi-core CPU, a single GPU or a cluster of GPUs, or multiple processors arranged for parallel processing) configured to execute instructions, which can be stored in a computer-readable medium and, when executed, cause the processor to perform a machine-trained algorithm for evaluating images. The ultrasound system can include a display or graphics processor operable to arrange ultrasound images and / or additional graphical information that can include annotations, confidence metrics, user instructions, tissue information, patient information, indicators, and other graphical components in a display window for display on a user interface of the ultrasound system. In some embodiments, the ultrasound images and associated measurements can be provided to a storage device and / or memory device, such as a picture archiving and communication system (PACS) for reporting purposes or future machine training.

[0022] Figure 1An example ultrasound system according to the principles of the present disclosure is shown. Ultrasound system 100 can include an ultrasound data acquisition unit 110. Ultrasound data acquisition unit 110 can include an ultrasound probe including an ultrasound sensor array 112 configured to transmit ultrasound pulses 114 into a region 116 (e.g., an abdomen) of a subject and receive ultrasound echoes 118 in response to the transmitted pulses. Region 116 can include a developing fetus (as shown) or various other anatomical objects such as a heart, a lung, or an umbilical cord. As further shown, ultrasound data acquisition unit 110 can include a beamformer 120, a transmit controller 121, and a signal processor 122. Transmit controller 121 can be configured to control transmission of ultrasound beams from sensor array 112. An image acquisition feature that can be controlled by transmit controller 121 is the imaging mode implemented by sensor array 112, e.g., 2D or 3D. For example, under the direction of transmit controller 121, beamformer 120 and sensor array 112 can switch from 2D imaging to 3D imaging after acquiring a 2D image of a region of interest. Signal processor 122 can be configured to generate a succession of discrete ultrasound image frames 124 from ultrasound echoes 118 received at array 112.

[0023] Image frames 124 can be passed to a data processor 126, e.g., a computing module or circuit. Data processor 126 can be configured to analyze image frames 124 by implementing various image segmentation and / or boundary detection techniques. Additionally or alternatively, data processor 126 can be configured to implement one or more neural networks trained to identify various anatomical features and / or generate user instructions for manipulating ultrasound transducer 112. In the illustrated embodiment, data processor 126 can be configured to implement a first neural network 128, a second neural network 130, and / or a third neural network 132. First neural network 128 (or multiple networks, as further described below) can be trained to identify one or more anatomical features visible within image frames 124 and, based on the identified anatomical features, generate instructions for manipulating ultrasound sensor array 112 in a manner needed to obtain an image of a target feature 117 (e.g., a face of an unborn infant). Second neural network 130 can be trained to identify a region of interest within an image of target feature 117, which can trigger sensor array 112 to acquire a 3D image of the region of interest under the direction of transmit controller 121. Third neural network 132 can be trained to perform one or more post-acquisition processing steps needed to generate a desired 3D portrait of the region of interest, e.g., application of artificial lighting. In various examples, data processor 126 can also be communicatively or otherwise coupled to a memory or database 134 configured to store various data types, including training data and newly acquired patient-specific data.

[0024] The system 100 can also include a display processor 136, e.g., a computational module or circuit, communicatively coupled with the data processor 126. The display processor 136 is also coupled with a user interface 138, such that the display processor 136 can link the data processor 126 (and thus any neural network(s) operating thereon) to the user interface 138, enabling the neural network output (e.g., user instructions in the form of motion control commands) to be displayed on the user interface 138. In embodiments, the display processor 136 can be configured to generate 2D ultrasound images 140 from the image frames 124 received at the data processor 126, which can then be displayed 2D in real-time via the user interface 138 as the ultrasound scan is being performed. In some examples, the display processor 136 can be configured to generate and display (via the user interface 138) one or more illustrative volume images of the target feature, each illustrative volume image corresponding to a particular view of the target image. The illustrative volume images can be selected by the user, prompting the system 100 to generate and display one or more commands for acquiring and generating actual volume images according to the user-selected views. A particular region of interest (“ROI”) 142, e.g., a ROI box, can also be displayed. In some examples, the ROI 142 can be located and trimmed by the second neural network 130. One or more notifications 144 (e.g., user instructions and / or alerts) can be overlaid on or displayed near the images 140 during the ultrasound scan. The user interface 138 can also be configured to receive user input 146 at any time before, during, or after the ultrasound scan. For example, the user interface 138 can be interactive, receiving user input 146 indicating a desired viewpoint for imaging the target feature 117 and / or indicating confirmation that imaging instructions have been followed. The user interface 138 can also be configured to display 3D images 148 acquired and processed by the ultrasound data acquisition unit 110, the data processor 126, and the display processor 136. The user interface 138 can include a display positioned external to the data processor 126, e.g., including a standalone display, augmented reality glasses, or a mobile phone.

[0025] Figure 1The configuration of the illustrated components can vary. For example, the system 100 can be portable or fixed. Various portable devices (e.g., laptop computers, tablet computers, smartphones, etc.) can be used to implement one or more functions of the system 100. In examples incorporating such devices, the ultrasound sensor array 112 may, for example, be connectable via a USB interface. In some embodiments, various components can be combined. For example, the data processor 126 can be merged with the display processor 136 and / or the user interface 138. For example, the first neural network 128, the second neural network 130, and / or the third neural network 132 can be merged such that the networks constitute subcomponents of a larger hierarchical network. In embodiments including separate discrete neural networks, the networks can be operatively cascaded arranged such that the output of the first neural network 128 comprises the input for the second neural network 130, and the output of the second neural network 130 comprises the input for the third neural network 132.

[0026] The ultrasound data acquisition unit 110 can be configured to acquire ultrasound data from one or more regions of interest 116, which can include the fetus and its features. The ultrasound sensor array 112 can include at least one transducer array configured to transmit and receive ultrasound energy. The setup of the ultrasound sensor array 112 can be adjustable during a scan. For example, under the direction of the beamformer 120 and the transmit controller 121, the ultrasound sensor array 112 can be configured to automatically (i.e., without user input) switch between 2D imaging mode and 3D imaging mode. In particular embodiments, the ultrasound sensor array 112 can be configured to switch to 3D imaging mode (or 4D or STIC mode) after identifying the target feature 117 in the 2D image 140 and delineating the ROI 142. Once in 3D imaging mode, the sensor array 112 can initiate an automatic sweep through the target feature 117, thereby acquiring a 3D volume of image data. Various transducer arrays can be used, such as linear arrays, convex arrays, or phased arrays. In different examples, the number and arrangement of transducer elements included in the sensor array 112 can vary. For example, the ultrasound sensor array 112 can include a 2D array of transducer elements, such as a matrix array probe. The 2D matrix array can be configured to electronically scan (via phased array beamforming) in both the elevation and azimuth dimensions for alternating 2D and 3D imaging. In addition to B-mode imaging, imaging modalities implemented in accordance with the disclosure herein can also include, for example, shear wave and / or Doppler. Various users can handle and operate the ultrasound data acquisition unit 110 to perform the methods described herein. In some examples, the user can be an inexperienced novice sonographer.

[0027] The data acquisition unit 110 can also include a beamformer 120 coupled to the ultrasound sensor array 112, for example including a microbeamformer or a combination of a microbeamformer and a main beamformer. The beamformer 120 can control the transmission of ultrasound energy, for example by forming ultrasound pulses into focused beams. The beamformer 120 can also be configured to control the reception of ultrasound signals so that discernible image data can be produced and processed with the aid of other system components. The role of the beamformer 120 can vary in different ultrasound probe variants. In some embodiments, the beamformer 120 can include two independent beamformers: a transmit beamformer configured to receive and process a sequence of pulses of ultrasound energy for transmission into the subject, and a separate receive beamformer configured to amplify, delay, and / or sum received ultrasound echo signals. In some embodiments, the beamformer 120 can include a microbeamformer coupled to a main beamformer, the microbeamformer operating on groups of sensor elements for both transmit and receive beamforming, the main beamformer operating on group inputs and outputs for both transmit and receive beamforming, respectively.

[0028] The transmit controller 121 (e.g., a computing module or circuit) can be communicatively, operatively, and / or physically coupled with the sensor array 112, the beamformer 120, the signal processor 122, the data processor 126, the display processor 136, and / or the user interface 138. In some examples, the transmit controller 121 can cause the sensor array 112 to switch to the 3D imaging mode via the transmit controller 121 upon receiving a user input that directs the switch, in response to the user input 146. In other examples, the transmit controller 121 can automatically initiate the switch, for example in response to receiving an indication from the data processor 126 that an ROI 142 has been identified in the 2D image 140 of the target feature 117.

[0029] The signal processor 122 (e.g., a computing module or circuit) can be communicatively, operatively, and / or physically coupled with the sensor array 112, the beamformer 120, and / or the transmit controller 121. In some examples, the signal processor 122 can be configured to process ultrasound signals received from the sensor array 112 to produce 2D image data 140 for display on the display 134. Figure 1In the illustrated example, the signal processor 122 is included as an integrated component of the data acquisition unit 110, but in other examples, the signal processor 122 can be a separate component. In some examples, the signal processor 122 can be housed with the sensor array 112, or it can be physically separate from the sensor array 112 but communicatively coupled (e.g., via a wired or wireless connection). The signal processor 122 can be configured to receive unfiltered and unordered ultrasound data embodying the ultrasound echoes 118 received at the sensor array 112. From this data, the signal processor 122 can continuously generate a plurality of ultrasound image frames 124 as the user scans the region 116.

[0030] In particular embodiments, the first neural network 128 can include a convolutional neural network trained to identify the presence of one or more anatomical features present in 2D ultrasound images (e.g., B-mode images), and in some examples, the orientation. Based on this determination, the first neural network 128 can generate one or more user instructions for manipulating the sensor array 112 in a manner required to acquire an image of another anatomical feature (such as the face of an unborn baby) from a particular vantage point that can be specified by the user. The instructions can be displayed sequentially in real-time as the user adjusts the sensor array 112 according to each instruction. The first neural network 128 can be configured to identify the anatomical feature as the user moves the sensor array 112 and generate new instructions accordingly, each new instruction based on image data present in each ultrasound image generated as the sensor array 112 is moved, e.g., embodying anatomical landmarks. In some examples, the user can confirm that an instruction has been implemented, e.g., via manual input 146 received at the user interface 138, signaling to the system that the next instruction can be displayed. The instructions can include directional commands to move, tilt, or rotate the sensor array 112 in a particular direction.

[0031] In some examples, the first neural network 128 can be trained to align the transducer array 112 to a target image plane based on previously acquired images stored in the database 134. The previously acquired images can be stored in association with motion control parameters and / or labels or scores for qualifying or validating newly acquired images, for example as described in PCT / EP2019 / 056072 and PCT / EP2019 / 056108, both of which are incorporated herein in their entirety by reference. Thus, one or more processors (e.g., data processor 126) can be configured to apply the first neural network 128 in a clinical setting to determine a directional command for aligning the transducer array 112 to a patient in a manner required by a user to acquire an image of a target feature. A new motion control command can be generated each time the transducer array 112 is repositioned. In some examples, the first neural network 128 can also be configured to predict whether a candidate for a motion control configuration (e.g., a control for changing an imaging plane within a volumetric ultrasound image) for repositioning the ultrasound transducer 112 will result in an optimal imaging position for a particular target view given an input image. The processor can then output the directional command to the display 138 in the format of an instruction and / or a visual indicator, and the user can manually align the transducer array 112 to the patient based on the instruction.

[0032] Additionally or alternatively, the generation of motion control parameters that are ultimately displayed to the user as instructions can be implemented via a plurality of neural networks or neural network layers, for example as specifically described in PCT / EP2019 / 056108. According to such examples, the first neural network 128 can include networks 128a, 128b, and 128c, where network 128a includes a prediction network trained to receive a currently acquired image and to speculate or infer a motion vector having the highest probability of reaching a desired position for capturing a target image view. Network 128b can include a fine-tuning network trained to verify whether a pair of images has the same level of quality or to select an image having a higher level of quality from the pair. Network 128c can include a target network trained to determine whether a target image view has been captured.

[0033] In some embodiments, the identification of the presence and orientation of one or more anatomical features present in the ultrasound image can not be performed by the first neural network. Such embodiments can involve the implementation of one or more boundary detection or image segmentation techniques by one or more processors of the system 100, for example data processor 126.

[0034] The second neural network 130 can include a convolutional neural network trained to define a ROI 142 within a 2D image of the target feature 117. As such, the second neural network 130 can be configured to operate after the first neural network 128 has successfully guided the user to capture an image 140 of the target feature 117. Defining the ROI 142 can involve placing a geometric shape (e.g., a box) within the image 140 of the target feature 117 and setting its size such that all non-targeted features (e.g., placenta, legs, arms, neck, etc.) are excluded from the ROI 142. In some examples, the ROI 142 can not include a geometric shape, but rather include a best-fit line positioned around salient features of the target feature 117 (e.g., nose, forehead, eyes, chin, ears, etc.). By defining the ROI 142 prior to capturing the 3D image, 3D data can be collected only within the ROI, which can improve the speed and efficiency of the system 100, as well as the quality of the resulting 3D image 148.

[0035] The third neural network 132 can include a convolutional neural network trained to perform one or more post-capture processing steps required to generate the 3D panoramic image 148 of the ROI 142. Example post-capture processing steps can include applying artificial lighting to the image such that the image includes shading. The direction of the artificial lighting applied can be adjusted by the third neural network 132 automatically or in response to user input 146. The third neural network 132 can also retouch the image to remove one or more undesirable features. Additionally or alternatively, the third neural network 132 can be configured to alter the imaging contrast such that certain features are highlighted or darkened. The modifications introduced by the third neural network 132 can be based at least in part on artistic quality. For example, the lighting, contrast, and / or retouching adjustments applied by the third neural network 132 can be designed to improve the aesthetic appearance of the 3D portrait generated by the system.

[0036] In some embodiments, one or more of the post-capture steps, such as the application of artificial lighting, can be implemented without a neural network. For example, one or more processors of the system 100, such as the data processor 126 or the display processor 136, can be configured to render the 3D image 148 of the ROI 142 in relation to a lighting model space such that lighting and shading regions of the anatomical features depicted within the ROI 142 are displayed according to stored settings of the system 100, as disclosed in, for example, US 2017 / 0119354, which is incorporated herein by reference in its entirety. The stored settings for the lighting model can be implemented automatically by the system 100, or the stored settings can be customizable or selectable from a plurality of setting options.

[0037] Each neural network 128, 130, and 132 can be implemented at least partially in a computer-readable medium comprising executable instructions that, when executed by a processor (e.g., data processor 126), can cause the processor to perform a machine-trained algorithm. For example, data processor 126 can be caused to perform a machine-trained algorithm to determine the presence and / or type of anatomical feature contained in an image frame based on acquired echo signals embodied therein. Data processor 126 can also be caused to perform a separate machine-trained algorithm to define the location, size, and / or shape of an ROI within an image frame, such as a 2D image frame of an image containing the face of an unborn infant. Another machine-trained algorithm implemented by data processor 126 can apply at least one image display setting configured to add shading and / or contrast to a 3D image.

[0038] To train each neural network 128, 130, and 132, a training set comprising multiple instances of input arrays and output classifications can be presented to the training algorithm(s) (e.g., an Alexnet training algorithm, as described by Krizhevsky, A., Sutskever, I., and Hinton, G. E., “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS 2012 or descendants thereof) of each network. The first neural network 128 can be trained using a large clinical database of ultrasound images obtained during prenatal ultrasound scans. The images can include fetuses at various stages of development from various imaging angles and positions. Thousands or even millions of training data sets can be presented to the neural network training algorithm associated with the first neural network 128 in order to train the neural network to recognize anatomical features and determine the ultrasound probe adjustments needed to acquire images of the target features based on the presence of the recognized features. In various examples, the number of ultrasound images used to train the first neural network 128 can range from approximately 50,000 to 200,000 or more. The number of images used to train the first neural network 128 can be increased if a larger number of different anatomical features are to be recognized. The number of training images can differ for different anatomical features and can depend on the variability of the appearance of certain features. For example, certain features can appear more consistently at certain stages of prenatal development than others. Training the first neural network 128 to recognize features with moderate to high variability can require more training images. The first neural network 128 can also be trained with clinically validated instructions for manipulating the ultrasound probe, each instruction associated with an initial set of anatomical features present in the current ultrasound image and a target anatomical feature to view from a particular vantage point. Thus, the first neural network 128 can be trained to recognize certain anatomical features present within a given ultrasound image and associate such features with one or more instructions needed to acquire images of the target features from the vantage point.

[0039] The second neural network 130 can also be trained using a large clinical database of ultrasound images obtained during prenatal ultrasound scans. Each of the training images can contain a defined ROI that can include a boundary of a face of an unborn baby. Thousands or even millions of training data sets can be presented to a neural network training algorithm associated with the second neural network 130 in order to train the neural network to define a ROI within any given 2D ultrasound image. In various examples, the number of ultrasound images used to train the second neural network 130 can be in the range of about 50,000 to 200,000 or more. The number of training images can be increased for a greater number of viewing options. For example, if a target feature can only be imaged from one direction, the number of training images can be less compared to embodiments in which a target feature can be imaged from multiple directions.

[0040] The third neural network 132 can also be trained using a large clinical database of ultrasound images obtained during prenatal ultrasound scans. Each of the images can contain a 3D image of a target feature with one or more post-acquisition settings applied. For example, just to name a few, each image can include artificial lighting, pixel contrast adjustment, and / or feature finishing modifications so that the third neural network 132 can learn which adjustments to apply to different images to ensure that a recognizable image of a target feature (e.g., a baby face) is generated according to common artistic preferences. The number of ultrasound images used to train the third neural network 132 can be in the range of about 50,000 to 200,000 or more. The number of training images can be increased to accommodate a greater number of post-acquisition modifications.

[0041] Figure 2 is a display of an illustrative volume image of a target feature generated by one or more processors described herein. The illustrative volume image is presented to a user on a user interface 238 as an optional option along with a user instruction 244 to“pick a desired 3D baby face view.” In the illustrated example, the target feature includes a face of an unborn baby. The user can select option A, B, or C that respectively correspond to a right profile view, a center front view, and a left profile view of the baby face. As shown, selection of option B can trigger one or more processors operating on the system that can be configured to implement one or more neural networks coupled with the user interface 238 to generate a user instruction for steering an ultrasound transducer to a target position in order to acquire and generate at least one image frame according to the particular view embodied by option B.

[0042] Figure 3is a display of user instructions 344 for ultrasound probe manipulation overlaid on a live 2D image 340 of a fetus displayed on the user interface 338. User instructions for ultrasound probe manipulation, such as user instructions 344, can be generated by the data processor 126, implementing one or more neural networks and / or image segmentation techniques, and output to the user interface 138 for display.

[0043] With respect to the ultrasound probe used to acquire the image 340, the user instructions 344 direct the user to “please press guided translation.” The exact language of the user instructions 344 can vary. In some examples, the instructions can include only one or more symbols, e.g., arrows. In further examples, the instructions can not be displayed visually at all, but can be conveyed as an audio prompt. The user instructions 344 are generated based on fetal anatomical landmarks identified by a neural network operating within the ultrasound system and also based on a viewing option selected by the user, as shown in Figure 2

[0044] In additional implementations, the visual indicator embodying the user instructions can include a graphical representation or view of the ultrasound probe, including one or more features of the probe, such as a handle, one or more adjustable knobs, and / or switches. According to such examples, the user instructions can include instructions to indicate the direction and / or amount to dial a knob, to turn on or off a switch, and / or the direction and / or degree to rotate the probe. In some examples, the motion control commands can include controls for operating the probe (or other imaging device) to change the imaging plane within the volumetric ultrasound image other than by changing the physical position of the probe. Generally, the user instructions can include motion control commands embodying any measurable data related to a particular position or motion of the imaging device (e.g., ultrasound transducer), as further described in PCT / EP2019 / 056072 and PCT / EP2019 / 056108.

[0045] Figure 4 is a display of another user instruction 444 for ultrasound probe manipulation overlaid on a live 2D image 440 of a fetus displayed on the user interface 438. At this stage, the user instruction 444 directs the user to “please press guided rotation.” The user instruction 444 can be generated based on fetal anatomical landmarks identified by one or more processors in the image 440 and also based on a viewing option selected by the user, as shown in Figure 2

[0046] Figure 5 ​​This is a display of another user instruction 544 for ultrasound probe manipulation, superimposed on a live 2D image 540 of the fetus displayed on the user interface 538. At this stage, the target feature 517 (i.e., the infant's face) is shown in the view along with the initial ROI 541. User instruction 544 informs the user that "optimal view acquired," and thus guides the user to "remain still." User instruction 544 can be generated based on facial landmarks recognized in image 540 by a neural network or other processing unit and also based on viewing options selected by the user, such as... Figure 2 As shown.

[0047] Figure 6 This includes the automatic localization of the refined ROI 642 and the display of a user notification 644 indicating the imminent start of 3D sweeping. In embodiments characterized by multiple discrete networks, individual neural networks can be configured to distinguish facial features from non-facial features and to ( Figure 5 The initial ROI 541 is refined to include only facial features for identification and definition of a refined ROI 642. Alternatively or additionally, facial features can be identified and refined via image segmentation. After defining the refined ROI 642, the system can perform a 3D sweep of the target features 617 within the refined ROI 642. The user interface 638 can prompt the user to confirm whether a sweep should be performed, or can warn the user that a sweep will be performed automatically, for example, at the end of a countdown. Therefore, the user notification 644 can notify the user "Automatically locate the ROI. Sweep into 3…2…1…".

[0048] Figure 7 It is in ( Figure 6 The display of the actual volumetric image of the target features captured in the refined ROI 642. In the example shown, the actual image includes the acquired 3D image 748 of the refined ROI 642 presented on the user interface 738, and a user notification 744 asking "Do you want another view?". As shown, the 3D image 748 includes shadows generated by artificial light sources, which may be applied by another processor and / or neural network operating on the system. If the user indicates that they desire another view, the user interface may, for example, display the actual volumetric image of the target features captured in the refined ROI 642. Figure 2 The initial viewing options are displayed in the same or similar manner as shown.

[0049] For example, some 3D images may be unavailable due to the fetus's position. In such cases, lower-layer neural networks (such as those in...) Figure 1 The first neural network 128 described herein can be configured to identify which views will produce high-quality images of the target features. (One or more) such views can be displayed on a user interface for the user to select, thereby initiating the display of user instructions required for image acquisition, such as... Figures 2-7 As shown.

[0050] Figure 8 is a flowchart of a method of ultrasound imaging performed in accordance with the principles of the present disclosure. The example method 800 illustrates steps that can be utilized by the systems and / or apparatuses described herein for acquiring 3D images of a target feature, such as the face of an unborn baby, in any order, which can be performed by a novice user following instructions generated by the system. The method 800 can be performed by an ultrasound imaging system, such as the system 100, or other systems, including, for example, mobile systems, such as LUMIFY by Koninklijke Philips N.V. (“Philips”). Additional example systems can include SPARQ and / or EPIQ, also produced by Philips.

[0051] In the illustrated embodiment, the method 800 begins at block 802 with “acquiring echo signals responsive to ultrasound pulses transmitted toward a target region.”

[0052] At block 804, the method involves “presenting to a user one or more illustrative volume images of the target feature, each illustrative volume image corresponding to a particular view of the target image.”

[0053] At block 806, the method involves “receiving a user selection of one of the illustrative volume images.”

[0054] At block 808, the method involves “generating two-dimensional (2D) image frames from the acquired echo signals of the target region.”

[0055] At block 810, the method involves “identifying one or more anatomical landmarks in the generated 2D image frames that correspond to the target feature.”

[0056] At block 812, the method involves “based on the anatomical landmarks and the particular view of the user-selected volume image, providing instructions for steering an ultrasound transducer to a target position for generating at least one 2D image frame specific to the particular view of the user-selected volume image.”

[0057] At block 814, the method involves “causing the ultrasound transducer to acquire additional echo signals at the target position.”

[0058] At block 816, the method involves “generating an actual volume image of the target feature and corresponding to the particular view of the user-selected volume image utilizing the acquired additional echo signals.”

[0059] In various embodiments in which components, systems and / or methods are implemented using programmable devices such as computer-based systems or programmable logic devices, it should be appreciated that the above systems and methods can be implemented using various known or later developed programming languages such as "C", "C++", "FORTRAN", "Pascal", "VHDL", etc. Accordingly, various storage media can be prepared which can contain information which can direct a device such as a computer to implement the above systems and / or methods. Once the appropriate device accesses the information and programs contained on the storage media, the storage media can provide the information and programs to the device, thereby enabling the device to perform the functions of the systems and / or methods described herein. For example, if a computer disk containing the appropriate material (such as source files, object files, executable files, etc.) is provided to a computer, the computer can receive the information, configure itself appropriately and perform the functions of the various systems and methods outlined in the above figures and flow charts to implement the various functions. That is, the computer can receive various portions of the information from the disk relating to different elements of the above systems and / or methods, implement the individual systems and / or methods and coordinate the functions of the individual systems and / or methods described above.

[0060] In view of the present disclosure, it should be noted that the various methods and devices described herein can be implemented in hardware, software and firmware. Moreover, the various methods and parameters are included by way of example only and not in any limiting sense. In view of the present disclosure, those of ordinary skill in the art can implement the present teachings in determining their own techniques and equipment needed to implement these techniques, while remaining within the scope of the present invention. The functions of one or more of the processors described herein can be incorporated into a fewer number of or a single processing unit (e.g., CPU), and can be implemented using application specific integrated circuits (ASICs) or general purpose processing circuitry programmed to perform the functions described herein in response to executable instructions.

[0061] While the present system can have been described with particular reference to an ultrasound imaging system, it is contemplated that the present system can be extended to other medical imaging systems in which one or more images are obtained in a systematic manner. Thus, the present system can be used to obtain and / or record image information related to, but not limited to, kidneys, testicles, breasts, ovaries, uterus, thyroid, liver, lungs, musculoskeletal, spleen, heart, arteries and vascular system, as well as other imaging applications related to ultrasound-guided interventions. Moreover, the present system can also include one or more procedures that can be used with conventional imaging systems such that they can provide the features and advantages of the present system. Certain additional advantages and features of this disclosure can be apparent to those skilled in the art upon studying the disclosure, or can be learned by practice of the novel systems and methods of the disclosure. Another advantage of the present system and method can be that conventional medical imaging systems can be easily upgraded to incorporate the features and advantages of the present system, device and method.

[0062] Of course, it is to be appreciated that any of the examples, embodiments or processes described herein can be combined with one or more of the other examples, embodiments and / or processes, or split into separate devices or device portions, in accordance with the present system, device and method.

[0063] Finally, the above discussion is meant to be illustrative only of the present system and should not be construed as limiting the claims to any particular embodiment or group of embodiments. Thus, while the present system has been described in detail with reference to exemplary embodiments, it should be appreciated that numerous modifications and alternatives can be devised by those of ordinary skill in the art without departing from the broader and intended spirit and scope of the present system as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present claims.

Claims

1. An ultrasound imaging system, comprising: An ultrasonic transducer configured to acquire echo signals in response to ultrasonic pulses emitted toward a target area; One or more processors that communicate with the ultrasonic transducer and are configured to: Presenting the user with one or more illustrative volumetric images of the target features, each illustrative volumetric image corresponding to a specific view of the target image; Receive user selection of one of the illustrative volumetric images; Two-dimensional (2D) image frames are generated based on the echo signals collected from the target area. Identify one or more anatomical landmarks corresponding to the target features in the generated 2D image frames; Based on the anatomical landmarks and the specific view of the user-selected volume image, instructions are provided for manipulating the ultrasound transducer to a target position to generate at least one 2D image frame specific to the specific view of the user-selected volume image. This allows the ultrasonic transducer to acquire additional echo signals at the target location; A controller is configured to automatically switch the ultrasonic transducer from 2D imaging mode to volumetric imaging mode upon receiving an indication from the one or more processors that a region of interest has been defined within the 2D image frame, the 2D image frame being specific to the particular view of the volumetric image selected by the user. The one or more multiprocessors are further configured to use the acquired additional echo signals to generate an actual volume image of the target feature and corresponding to the specific view of the volume image selected by the user.

2. The ultrasound imaging system according to claim 1, wherein, The one or more processors are configured to identify the one or more anatomical landmarks via image segmentation.

3. The ultrasound imaging system according to claim 1, wherein, The one or more processors are configured to identify the one or more anatomical landmarks via implementation of a neural network trained to recognize the anatomical landmarks.

4. The ultrasound imaging system according to claim 1, wherein, The one or more processors are also configured to apply an artificial light source to the actual volume image based on the specific view.

5. The ultrasound imaging system according to claim 1, wherein, The target features include the face of an unborn infant.

6. The ultrasound imaging system according to claim 1, wherein, The one or more processors are configured to generate instructions for manipulating the ultrasonic transducer by inputting the 2D image frames into an artificial neural network trained to compare the 2D image frames with stored image frames embodying the target features.

7. The ultrasound imaging system according to claim 6, wherein, The artificial neural network is configured to generate new instructions for manipulating the ultrasonic transducer in the event of repositioning of the ultrasonic transducer.

8. The ultrasound imaging system of claim 1, further comprising a user interface communicatively coupled to the one or more processors and configured to display the instructions for manipulating the ultrasound transducer to the target location.

9. A method for ultrasound imaging, the method comprising: Acquire echo signals in response to ultrasonic pulses emitted toward the target area; Presenting the user with one or more illustrative volumetric images of the target features, each illustrative volumetric image corresponding to a specific view of the target image; Receive user selection of one of the illustrative volumetric images; Two-dimensional (2D) image frames are generated based on the echo signals collected from the target area. Identify one or more anatomical landmarks corresponding to the target features in the generated 2D image frames; Based on the anatomical landmarks and the specific view of the user-selected volume image, instructions are provided for manipulating an ultrasound transducer to a target location to generate at least one 2D image frame specific to the specific view of the user-selected volume image. This allows the ultrasonic transducer to acquire additional echo signals at the target location; Upon receiving an indication that a region of interest has been defined within the 2D image frame, the ultrasonic transducer is automatically switched from 2D imaging mode to volumetric imaging mode, the 2D image frame being specific to the particular view of the volumetric image selected by the user. and The acquired additional echo signal is used to generate an actual volume image of the target feature and corresponding to the specific view of the volume image selected by the user.

10. The method of claim 9, further comprising applying an artificial light source, image contrast adjustment, or both to the actual volume image.

11. The method according to claim 9, wherein, The target features include the face of an unborn infant.

12. The method according to claim 9, wherein, Identifying the one or more anatomical landmarks involves image segmentation or implementation of at least one neural network trained to identify the anatomical landmarks.

13. The method of claim 9, further comprising displaying the instructions for manipulating the ultrasonic transducer.

14. The method of claim 9, further comprising: Additional anatomical landmarks of the target feature are identified while manipulating the ultrasound transducer; and Additional instructions for manipulating the ultrasonic transducer are generated based on the additional anatomical landmarks identified during manipulation of the ultrasonic transducer.

15. A non-transient computer-readable medium comprising executable instructions, which, when executed, cause a processor of a medical imaging system to perform any one of the methods according to claims 9-14.

Images