Systems and methods for directing the capture of supersonic images - Patents.com

Abstract

To perform orientation for correct use of a mobile handheld ultrasound device.SOLUTION: A downloadable navigator for a mobile ultrasound unit having an ultrasound probe is implemented on a mobile computing device. The navigator includes: a trained orientation neural network to receive a non-canonical image of a body part from the mobile ultrasound unit and to generate a transformation associated with the non-canonical image, the transformation transforming from a position and rotation associated with a canonical image to a position and rotation associated with the non-canonical image; and a result converter to convert the transformation into orientation instructions for a user of the probe and to provide and display the orientation instructions to the user to change the position and rotation of the probe.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates generally to portable handheld ultrasound devices and, more particularly, to orientation for proper use.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 671,692, filed May 15, 2018, which is incorporated herein by reference. [Background technology]

[0003] Medical ultrasound (also known as diagnostic ultrasound or sonography) is a diagnostic imaging technique based on the application of ultrasound. It is used to create images of internal body structures such as tendons, muscles, joints, blood vessels, and internal organs.

[0004] To obtain accurate images for effective examination and diagnosis, the ultrasound transducer must be positioned at an angular position relative to the relevant organ or body part in space, as shown in Figure 1. Figure 1 illustrates an ultrasound image of an organ of interest 12 taken with a transducer 14. It will be appreciated that the skill of navigating the transducer 14 to the precise angular position required to obtain an optimal or "standard" image of the organ 12 is crucial to a successful ultrasound examination. This process typically requires a trained and skilled sonographer.

[0005] For example, to perform an echocardiogram, a sonographer needs to take images of the heart from various standard views, such as the four-chamber view, the two-chamber view, etc. Correct positioning of the transducer is crucial to receiving an optimal view of the left ventricle and thereby extracting functional information about the heart.

[0006] Mobile ultrasound machines or devices are known in the art, such as Lumify, available from Philips, Inc. These mobile ultrasound machines are available in the form of a transducer that communicates with a program that can be downloaded to any portable handheld device, such as a smartphone or tablet.

[0007] The availability of such devices means that ultrasound examinations can be performed off-site, for example as a triage tool in ambulances, on the battlefield, in urgent care facilities, nursing homes, etc., without the need for bulky and expensive equipment. Summary of the Invention

[0008] In accordance with a preferred embodiment of the present invention, there is provided a downloadable navigator for a mobile ultrasound unit having an ultrasound probe, the navigator being implemented on a portable computing device, the navigator including: a trained orientation neural network that receives a non-canonical image of a body part from the mobile ultrasound unit and generates a transformation associated with the non-canonical image, the transformation converting from a position and rotation associated with a canonical image to a position and rotation associated with the non-canonical image; and a result converter that converts the transformation into orientation instructions for a user of the probe and provides and displays the orientation instructions to the user for changing the position and rotation of the probe.

[0009] Furthermore, in accordance with a preferred embodiment of the present invention, the navigator further includes a trainer for training the orientation neural network using the standard image, non-standard images taken around the standard image, and transformations from the standard image to positions and rotations in space associated with the non-standard image.

[0010] Further in accordance with a preferred embodiment of the present invention, the trainer includes a training converter that receives IMU data during a training session from an IMU (Inertial Measurement Unit) attached to a training probe, the IMU data providing positions and rotations associated with the non-standard image and the standard image, and transforms those positions and rotations from those associated with the standard image to those associated with the non-standard image.

[0011] Furthermore, in accordance with a preferred embodiment of the present invention, the trainer includes an untrained oriented neural network and a loss function for training the untrained oriented neural network, the loss function reducing the distance between a computed transformation produced by the untrained oriented neural network and a ground truth transformation of each non-standard image.

[0012] Furthermore, in accordance with a preferred embodiment of the present invention, the loss function further comprises a probability of constraining the computed transformation to one of a plurality of different standard orientations.

[0013] Furthermore, in accordance with a preferred embodiment of the present invention, the standard image is one of a plurality of standard images.

[0014] Further in accordance with a preferred embodiment of the present invention, the navigator includes a diagnoser for making a diagnosis from final images produced by the probe when viewing the standard images.

[0015] Furthermore, in accordance with a preferred embodiment of the present invention, the portable computing device is one of a smartphone, a tablet, a laptop, a personal computer, and a smart appliance.

[0016] Further in accordance with a preferred embodiment of the present invention, the navigator includes a set creator that receives a plurality of transformations from the trained orientation neural network in response to images from the probe and generates a set of images and their associated transformations, a sufficiency checker that determines when a sufficient set has been generated, and a trained cyclical canonical view neural network for generating a set of summary cyclical canonical images that show changes in the body part during its cycle.

[0017] Furthermore, in accordance with a preferred embodiment of the present invention, the navigator further includes a cyclical canonical view trainer for training an untrained cyclical canonical view neural network with the set of images, their associated transformations, and their associated summarized cyclical canonical images at each time point in a body cycle.

[0018] Furthermore, in accordance with a preferred embodiment of the present invention, the body-part cycle is a cardiac cycle.

[0019] Furthermore, in accordance with a preferred embodiment of the present invention, each set contains a single element.

[0020] In accordance with a preferred embodiment of the present invention, there is provided a navigator for a mobile ultrasound unit implemented on a portable computing device having an ultrasound probe, the navigator including: a trained orientation neural network for providing orientation information for a plurality of ultrasound images captured around a body part, the orientation information for orienting the images relative to a canonical view of the body part; and a volume reconstructor for orienting the images according to the orientation information, generating a volume representation of the body part from the oriented images using tomographic reconstruction, and generating a canonical image of the canonical view from the volume representation.

[0021] Further in accordance with a preferred embodiment of the present invention, the navigator includes a sufficiency checker for receiving directions from the trained orientation neural network in response to images from the probe and for determining when sufficient images have been received, and a result converter for requesting further images from the trained orientation neural network in response to the sufficiency checker.

[0022] Further in accordance with a preferred embodiment of the present invention the navigator includes a diagnoser for making a diagnosis from the volumetric representation of the body-part.

[0023] According to a preferred embodiment of the present invention, there is provided a navigator for a mobile ultrasound unit having an ultrasound probe implemented on a mobile device, the navigator including a trained mapping neural network for receiving a non-canonical image of a body part from the probe, mapping the non-canonical image to non-canonical map points on a displayable map, and mapping a plurality of canonical images associated with the non-canonical image to canonical map points on the displayable map, and a result converter for displaying a map marked with the canonical and non-canonical map points.

[0024] Furthermore, in accordance with a preferred embodiment of the present invention, the trained mapping neural network includes a loss function such that changes in the probe's motion produce small movements on the displayable map, the distances between the images are similar to the distances between positions on the map, and the optimal path from one standard image to another is a straight, constant-velocity trajectory.

[0025] Furthermore, in accordance with a preferred embodiment of the present invention, the navigator also includes a diagnoser that makes a diagnosis from a final image generated by the probe when a user moves the probe to one of the standard map points.

[0026] In accordance with a preferred embodiment of the present invention, there is provided a downloadable navigator for a mobile ultrasound unit having an ultrasound probe, the navigator being implemented on a mobile device, the navigator including a set creator that receives images from the probe over time and generates sets of images, a sufficiency checker that determines when a sufficient set has been generated, and a periodic canonical view neural network that generates a set of summarized periodic canonical images that show changes in the body part during its cycle.

[0027] Furthermore, in accordance with a preferred embodiment of the present invention, the navigator also includes a diagnoser for making a diagnosis from the final images generated by the periodic standard view neural network.

[0028] In accordance with a preferred embodiment of the present invention, there is provided a method for a mobile ultrasound unit having an ultrasound probe, the method being implemented on a portable computing device, the method comprising the steps of: receiving a non-standard image of a body part from the mobile ultrasound unit using a trained orientation neural network; generating a transformation associated with the non-standard image, the transformation converting from a position and rotation associated with a standard image to a position and rotation associated with the non-standard image; and translating the transformation into directional instructions for a user of the probe, and providing and displaying the directional instructions to the user for changing the position and rotation of the probe.

[0029] Further in accordance with a preferred embodiment of the present invention, the method includes training the orientation neural network using the standard image, non-standard images captured around the standard image, and a transformation from the standard image to a position and rotation in space associated with the non-standard image.

[0030] Further, in accordance with a preferred embodiment of the present invention, training includes receiving IMU data during a training session from an IMU (Inertial Measurement Unit) attached to a training probe, the IMU data providing positions and rotations associated with the non-standard image and the standard image, and transforming the positions and rotations from the positions and rotations associated with the standard image to the positions and rotations associated with the non-standard image.

[0031] Furthermore, in accordance with a preferred embodiment of the present invention, the trained mapping neural network includes a loss function such that changes in the probe's motion produce small movements on the displayable map, the distances between the images are similar to the distances between positions on the map, and the optimal path from one standard image to another is a straight, constant-velocity trajectory.

[0032] Furthermore, in accordance with a preferred embodiment of the present invention, the loss function further comprises a probability of constraining the computed transformation to one of a plurality of different standard orientations.

[0033] Furthermore, in accordance with a preferred embodiment of the present invention, the standard image is one of a plurality of standard images.

[0034] Further in accordance with a preferred embodiment of the present invention the method includes the step of making a diagnosis from a final image produced by the probe when viewing the standard image.

[0035] Furthermore, in accordance with a preferred embodiment of the present invention, the portable computing device is one of a smartphone, a tablet, a laptop, a personal computer, and a smart appliance.

[0036] Furthermore, in accordance with a preferred embodiment of the present invention, the method further includes the steps of receiving a plurality of transformations from the trained oriented neural network in response to images from the probe and generating a set of images and their associated transformations, determining when a sufficient set has been generated, and using the trained periodic standard view neural network to generate a set of summary periodic standard images showing changes in the body part during its cycle.

[0037] Furthermore, in accordance with a preferred embodiment of the present invention, the method further comprises the step of training an untrained periodic standard view neural network with the set of images, their associated transformations, and their associated summary periodic standard images at each time point in the body's cycle.

[0038] Furthermore in accordance with a preferred embodiment of the present invention the cycle of the body part is a cardiac cycle.

[0039] Furthermore, in accordance with a preferred embodiment of the present invention, each set contains a single element.

[0040] Further in accordance with a preferred embodiment of the present invention, there is provided a method for a mobile ultrasound unit implemented on a portable computing device having an ultrasound probe, the method including the steps of: using a trained orientation neural network to provide orientation information for a plurality of ultrasound images captured around a body part, the orientation information for orienting the images relative to a standard view of the body part; orienting the images according to the orientation information, generating a volumetric representation of the body part from the oriented images using tomographic reconstruction, and generating a standard image of a standard view from the volumetric representation.

[0041] Further in accordance with a preferred embodiment of the present invention, the method includes the steps of receiving directions from the trained orientation neural network in response to images from the probe to determine when sufficient images have been received, and requesting further images from the trained orientation neural network in response to the received directions.

[0042] Furthermore, in accordance with a preferred embodiment of the present invention, the method further comprises the step of making a diagnosis from the volumetric representation of the body-part.

[0043] Further in accordance with a preferred embodiment of the present invention, there is provided a method for a mobile ultrasound unit having an ultrasound probe implemented on a mobile device, the method including the steps of receiving a non-standard image of a body part from the probe using a trained mapping neural network, mapping the non-standard image to non-standard map points on a displayable map, mapping a plurality of standard images associated with the non-standard image to standard map points on the displayable map, and displaying the map marked with the standard and non-standard map points.

[0044] Furthermore, in accordance with a preferred embodiment of the present invention, the trained mapping neural network includes a loss function that causes changes in the motion of the probe to produce small movements on the displayable map, such that the distance between the images resembles a straight, constant velocity trajectory.

[0045] Furthermore, in accordance with a preferred embodiment of the present invention, the method further comprises the step of making a diagnosis from a final image produced by the probe when the user moves the probe to one of the standard map points.

[0046] Further in accordance with a preferred embodiment of the present invention, there is provided a method for a mobile ultrasound unit having an ultrasound probe implemented on a mobile device, the method comprising the steps of receiving images from said probe over time and generating a set of images, determining when a sufficient set has been generated, and generating, with a periodic canonical view neural network, a set of summarized periodic canonical images showing changes in said body part during its cycle.

[0047] Further in accordance with a preferred embodiment of the present invention the method includes the step of making a diagnosis from a final image produced by the periodic standard view neural network. [Brief explanation of the drawings]

[0048] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, and also as to objects, images and advantages, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram showing how ultrasound transducers are positioned to capture images of a body part. [Figure 2] FIG. 2 is a schematic diagram of an ultrasound navigator constructed and operative in accordance with the present invention. [Figure 3] 3A and 3B are schematic diagrams illustrating how the navigator of FIG. 2, constructed and operative in accordance with the present invention, assists a non-sonographer in orienting a probe and transducer to capture an appropriate image of a body part. [Figure 4] FIG. 4 is a schematic illustration of a transformation of training probe orientation for a non-standard image of an organ and its associated standard image, constructed and operative in accordance with the present invention. [Figure 5] FIG. 5 is a schematic diagram of a training process for an oriented neural network constructed and operative in accordance with the present invention. [Figure 6] FIG. 6 is a schematic diagram of elements of the navigator of FIG. 2, constructed and operative in accordance with the present invention. [Figure 7] FIG. 7 is a schematic diagram of elements of an alternative embodiment to the navigator of FIG. 2, constructed and operative in accordance with the present invention. [Figure 8] 8A, 8B and 8C are schematic diagrams of the elements and functionality of an alternative embodiment to the navigator of FIG. 2, constructed and operable in accordance with the present invention. [Figure 9] 9A and 9B are schematic illustrations of elements of an alternative embodiment to the navigator of FIG. 2 during training and operation, constructed and operative in accordance with the present invention. [Figure 10]10A and 10B are schematic diagrams of elements of an alternative embodiment to the navigator of FIGS. 9A and 9B during training and operation, constructed and operable in accordance with the present invention. It should be understood that for simplicity and clarity of illustration, the elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Furthermore, where considered appropriate, numerals may be repeated among the figures to indicate corresponding or similar elements. DETAILED DESCRIPTION OF THE INVENTION

[0049] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

[0050] Applicant understands that the availability of mobile ultrasound devices away from traditional locations such as hospitals means that these devices may be utilized by untrained or non-sonographers. However, untrained physicians, emergency medical personnel, or even patients themselves may not have the training or knowledge to properly perform these ultrasound examinations. It will be understood that different organs and body parts require different training.

[0051] Prior art systems, such as those described in U.S. Patent Application Publication No. 2018 / 0153505, published June 7, 2018, entitled "Guided Navigation of an Ultrasound Probe," and U.S. Patent Application Publication No. 2016 / 0143627, published May 26, 2016, entitled "Ultrasound Acquisition Feedback Guidance to a Target View," teach how to determine deviations between a provided image and a preferred standard image of a particular body part to assist a non-sonographer in guiding the transducer in an optimal direction to capture an optimal image.

[0052] Applicant realizes that these prior art systems do not provide a complete solution with respect to rotation calculations. Applicant also realizes that these prior art systems are not particularly useful because they require additional hardware (e.g., inertial measurement units such as magnetometers, gyroscopes, accelerometers, etc.) to assist non-sonographers in determining the position of the probe. Applicant realizes that a system that does not require additional hardware and is readily accessible, such as via download, for use as an integration or overlay with the processing software of an associated mobile ultrasound device would be far more useful. As a result, the present invention operates solely with digital images generated by the ultrasound unit.

[0053] Referring to FIG. 2, there is shown an ultrasound navigator 100 according to a first embodiment of the present invention, which can be downloaded from a mobile application store 10, such as Apple's Appstore or Google's Google Play, to any portable computing device, such as a smartphone, tablet, laptop, personal computer, smart appliance, etc.

[0054] It will be appreciated that the navigator 100 may include (as part of the download) a trained orientation neural network 15. The orientation neural network 15 is described in more detail herein below. As noted above, the navigator 100 may be integrated with or used as an overlay to the processing software of an associated mobile ultrasound device.

[0055] Thus, the user 5 can use a transducer or probe 7 (associated with a mobile ultrasound unit 8) on the patient 9 to provide images of the relevant body part to the navigator 100, and in response, the navigator 100 can provide directional instructions as to which direction to point the probe 7. It will be appreciated that the process can be iterative, with the non-sonographer or user 5 making multiple attempts to correctly point the probe 7 in order to receive an appropriate image. In accordance with a preferred embodiment of the present invention, even when the navigator 100 only receives images, the "directional" instructions can include both position (location in two-dimensional or three-dimensional space) and rotational information (rotation in 3D space).

[0056] Referring to Figures 3A and 3B, the navigator 100 is shown to assist a non-sonographer 5 in orienting a probe 7 to capture a good image of a particular body part. Figure 3A shows the probe 7, labeled 7A, in an incorrect position, meaning the resulting image, labeled 20A, is not standard. Figure 3A also includes a set of arrows 21 instructing the user 5 to change the rotation of the probe 7A. Arrow 21A indicates a "pitch up" rotation. Figure 3B shows the probe 7B in a newly pitched US orientation and the resulting image 20B, which still does not provide a standard image, but is better. Arrow 21B indicates that a new "yaw" rotation may be useful.

[0057] As described above, the navigator 100 may receive an orientation neural network 15, which may be trained with expert data taken by experienced sonographers of a particular body part or organ of interest. The received training data may include standard images of the particular body part and associated non-standard images, and for each, the orientation (i.e., position and rotation) of the sonographer's probe in space. It should be appreciated that this information may be generated using a probe with an associated IMU (inertial measurement unit, including magnetometers, gyroscopes, accelerometers, etc.). The IMU may measure the orientation of the probe when the image is captured.

[0058] Referring to Figure 4, the transformation between the orientation of a training probe 4c used by a trained sonographer to capture standard images and its orientation when capturing non-standard images of an organ is shown. The orientation of the training probe 4i when viewing the ith non-standard image is determined by a "frame of reference" in space F i where the reference frame F i The IMU can have six degrees of freedom (6DoF), corresponding to a three-axis system (Q) with three rotations about the axis and three translations along the axis, that the IMU can measure.

[0059] Reference Frame F i can refer to a reference frame at the origin O, which in the present invention may be the organ, and whose reference frame in space can be defined as F0. i For each of the i where there may be a transformation R c is expressed as follows: c , may be a transformation to a desired orientation for viewing the standard image, labeled as R c =F c F0 -1 Ri =F i F0 -1 (1) Here, F0 -1 is the inverse transformation of F0. Therefore, the transformation T from the standard pose to the i-th non-standard pose i is R i R c -1 : T i =R i R c -1 =F i F0 -1 (F0F c -1 )=F i F c -1 (2)

[0060] 5 illustrates the training process for the orientation neural network 15 using a trainer 30. An experienced sonographer 2 using a training probe 4 on a patient 3 can provide both standard and related non-standard images of a particular body part. The training probe 4 determines the orientation of the probe F when the images are captured. i It will be appreciated that the sensor may be associated with an IMU 6 (which may include a magnetometer, gyroscope, accelerometer, etc.) capable of measuring:

[0061] The training converter 22 generates orientation data F for each image. i , and transform T from the associated standard position as described above with respect to FIG. i =R i R c -1 Specifically, the training converter 22 can obtain images X from the training probe 4 and process them as needed. The database 20 can store non-standard images X i , and their direction data F i and their transformation data T i The database 20 can also store standard image Xc and their associated direction data F c It will be appreciated that there may be multiple standard images for a body part. For example, the heart may have a four-chamber standard image, a two-chamber standard image, etc., and thus the training converter 22 may store a transformation T for each associated standard image. i It will be appreciated that the relevant standard images may be provided manually or may be determined automatically by any suitable algorithm.

[0062] The input training data for trainer 30 is image X i and its associated ground truth transformation T i It will be appreciated that for each non-standard image, the trainer 30 can learn a position transformation of the probe 4 to transform from each viewed standard image to each viewed non-standard image. The input data can include data from many different patients 3, so that the trainer 30 can learn the position transformation of the image X, which may be due to the sex, age, weight, etc. of the patient 3 and other factors that affect the transformation information between the non-standard image and the standard image. i It will be understood that changes can be learned.

[0063] The trainer 30 may be any suitable neural network trainer, such as a convolutional neural network trainer, which trains the computed transformation S(X i ) and the associated image X from the standard image i The ground truth transformation T for i It will be further appreciated that the network can be trained by updating it to minimize the energy "loss" as determined by a loss function such as the distance between the transformation S(X i ) starts as an untrained neural network and ends as a trained neural network.

[0064] The distance function may be any suitable distance function. If there are multiple relevant standard images, the oriented neural network 15 may generate a ground truth transformation T for each non-standard image. i The loss function "Loss" can be calculated as follows: Loss=loss(S(X i ),T i ) (3)

[0065] Once trained, the oriented neural network 15 generates a i , a transformation T for the user probe 7 can be generated. This transformation can then be inverted or converted to direct the user 5 from a non-standard image orientation to a standard image orientation, as described in more detail below.

[0066] 6 shows the components of the navigator 100. The navigator 100 may include a trained oriented neural network 15, a results converter 40, and a diagnoser 50.

[0067] As described above, user 5 can randomly position user probe 7 relative to a desired body part. The trained orientation neural network 15 can provide a transformation T from the associated standard image of the particular body part to the current non-standard image. Result converter 40 can invert the generated transformation to provide orientation instructions for probe 7 from the current position and rotation at which the non-standard image is viewed to the position and rotation at which the associated standard image is viewed. Result converter 40 can provide and display these orientation instructions to user 5 in a variety of ways. It will be appreciated that this process can be repeated until user 5 positions probe 7 correctly (within error).

[0068] The result converter 40 can convert the orientation data S(X) generated by the trained orientation neural network 15 for the selected standard image into an orientation that is explainable to the user 5. Any suitable display can be utilized. An exemplary display is shown herein with reference to FIGS. 3A and 3B. It will be appreciated that the result converter 40 can use any suitable interface and can display (for example) colored rotation markings. Furthermore, the result converter 40 can include an element that allows the user 5 to indicate which standard image is currently of interest when multiple standard images of a body part exist.

[0069] Diagnosis device 50 can receive the final standard image generated by user 5 and detect abnormalities therein. Diagnosis device 50 can be any suitable diagnosis device. For example, diagnosis device 50 can perform the diagnosis method of International Publication No. WO 2018 / 136805, published July 26, 2018, which is assigned to the common assignee of the present invention and is incorporated herein by reference.

[0070] Applicant recognizes that the fact that there are multiple standard images for a single body part and that there are standard known movements from one standard image to another can be utilized to reduce errors in the output of the trained orientation neural network 15.

[0071] In this improved embodiment, the oriented neural network 15 may be trained on a number of standard images. i For example, for a pair of standard images c and c', there can be multiple computed transformations for the same image X i The transformation S calculated for c (X i ) and S c’ (X i ) that are related to the ground truth transformation Tc,i and T c’,i can have:

[0072] Furthermore, the known motion transformation T defined as k There is. T k =R c R c’ -1 (4) where R c is the standard image c, and R C’ is the standard image c'. These known motions are approximately constant across different objects, so the transformation T k is the calculated transformation S c (X i ) and S c’ (X i ) can be used to constrain the vector to one of the standard directions. To do this, we use the probability measure P k is added to the loss used to train the oriented neural network 15 as follows: k (S c (X i )S c’ (X i ) -1 ) can be used to define Loss=loss(S c (X i ),T c,i )+loss(S c’ (X i ), Tc ’,i )-δ * logP k (S c (X i )S c’ (X i ) -1 ) (5)

[0073] Probability measure P k is the ground truth transformation T between standard poses c and c' between different subjects. kFurthermore, there are multiple probability measures for each body part, one for each pair of standard images of the body part, and each probability measure P k However, we can define a separate additional term in the loss function.

[0074] In an alternative embodiment, the navigator labeled 100' may include a sufficiency checker 60 and a volume reconstructor 70, as shown in FIG.

[0075] The volume reconstructor 70 can utilize the output of the trained oriented neural network 15 to generate the image X generated by the probe 7. i From this, a 3D or 4D function and / or a 3D volume or a 3D spatiotemporal volume of the body part of interest can be generated. In this embodiment, the image X i can be considered as a cross section of the body part of interest.

[0076] Sufficiency checker 60 can check that enough cross sections have been received via trained orientation neural network 15 to perform 3D / 4D volume reconstruction and guide user 5 accordingly (via result converter 40). For example, sufficiency checker 60 can determine when a preset minimum number of images have been acquired.

[0077] Based on instructions from the sufficiency checker 60, the volume reconstructor 70 generates the 3D / 4D volume, after which the reconstructor 70 can derive relevant standard views from the generated volume and provide them to the diagnoser 50. It will be understood that the standard views in this embodiment are generated from the generated volume and may or may not be present in the images used to generate the volume.

[0078] The volume reconstructor 70 can reconstruct 3D / 4D functions and / or volumes from the images using tomographic reconstruction, such as those based on the inverse Radon transform or other means. It will be appreciated that knowledge of the location of the cross-sections in 3D space or 4D space-time is crucial for successful volume tomographic reconstruction. Applicant has determined that the trained directional neural network 15 can propose a transformation S(X) to the probe 7 for each image taken, and also to reconstruct the image X from a fixed 2D image plane. i When generating probe 4 i is located in 3D direction Q in space. i I understand that I can use a transformation S(X) to rotate the pixels of

[0079] The volume reconstructor 70 computes the volume for each image X i The transformation S(X i ) and applies this transformation to move the image (as output from the probe) from the image plane to the plane defined by the probe's transformation, resulting in a rotational cross-section of the body part, CS i The volume reconstructor 70 then uses tomographic reconstruction to generate the slices CS(X i ) from which the volume of the body part of interest can be constructed.

[0080] Conversion S(X i ), first, image X i is the 2D position (x j ,y j ) and intensity I j It will be appreciated that the volume reconstructor 70 includes a set of pixels having 3D pixel positions (x j ,y j ,0) to convert S(X i ) to image X i We then apply the operator H to generate an approximation of the 3D orientation Q of the oriented image X as follows: i can be centered or scaled.

[0081] Q=H * S(X i ) * [x j ,y j ,0] T (6)

[0082] The volume reconstructor 70 provides the generated standard images to the diagnoser 50, which may then provide a diagnosis from the standard images as described above.

[0083] In yet another embodiment illustrated in Figures 8A, 8B, and 8C, the navigator labeled 100'' may include an image mapping neural network 90. ​​The mapping neural network 90 may map each image X i can be mapped onto a 2D plane 92 (FIG. 8B). A , X B , X D is shown being mapped to three different locations A, B, and D on plane 92.

[0084] The result converter, labeled 42, can display a 2D plane 92 to the user 5, marking his current position as a dot of one color (e.g., gray (shown as a shaded dot in Figure 8C)) and the position of the standard image of this body part as a dot of another color (shown as numbered circles 1-5 in Figure 8C). Also shown in Figure 8C is a diagram of the acquired image X. i The map 92 is shown. The map point M(X i ) is a non-standard image on Map 92 X i The other numbered circles may be standard map points representing the desired or requested standard view c. The user 5 may use trial and error movements of the probe 7 to find the map points M(X i) can be fitted towards the desired circle and the mapper 90 can recreate the 2D plane 92 for each new image i from the probe 7.

[0085] Applicant has determined that small changes in the movement of the probe 7 produce small movements on the 2D plane 92, resulting in the image X i Applicant recognizes that the distance between should be similar to the distance between locations on a map. Furthermore, applicant recognizes that the optimal path from one standard image to another should be a straight, constant velocity trajectory.

[0086] In this embodiment, the mapping neural network 90 maps each image X i and its associated standard view image X c It will be appreciated that the input data may include:

[0087] The mapping neural network 90 calculates the calculated map points M(X i ) and each standard view C j The relevant map points M(X c ) can be incorporated as a loss function to minimize the distance between Loss=loss(M(X i ),M(X cj )) (7)

[0088] Image X to incorporate the best path to different standard views i is a probability vector p that defines how close is on the path to the jth desired standard image c. ij Then we can update the loss function to: Loss=loss(M(X i ),Σp ij M(X cj )) (8)

[0089] To preserve the distance, we can update the loss function as follows: TIFF0007811364000001.tif11170

[0090] It will be appreciated that plane 92 may be either a 2D plane or a 3D volume, as desired, and the mapping operations described above work equally well for mapping to a 3D volume.

[0091] Applicant recognizes that, with the appropriate type of training, neural networks can be trained to generate standard images in addition to generating transformation information. This can be particularly useful when input from non-sonographers is expected to be noisy (because their hands are not steady enough) and / or when standard views are desired to confirm that a body part is functioning. For example, sonographers routinely provide information about the complete cardiac cycle from systole to diastole and back to systole for cardiac function analysis.

[0092] 9A and 9B, the navigator 100 includes a periodic standard view neural network 110, which may be a neural network trained from the output of the trained oriented neural network 15. The standard view cycler 110 can repeatedly aggregate images to reduce noise and provide a less noisy summary of an organ cycle, such as (for example) the cardiac cycle.

[0093] As shown in FIG. 9A, the elements required to train a periodic standard view neural network 110 can include a trained orientation neural network 15, a set creator 112 for generating inputs to the network 110, and a periodic standard view trainer 115.

[0094] In this embodiment, a skilled sonographer 2 can provide multiple ultrasound images m taken over time and multiple images n taken over time at a single standard view pose c. The set creator 112 generates images X from a trained oriented neural network 15. m , and its associated transformation information S(X m ) and their associated standard view images X c,n A skilled sonographer 2 can provide such a correlation.

[0095] The set creator 112 then creates the triplet {[Y m ,Z m ],[W n ]}, where [Y m ,Z m ] is the input to the periodic standard view trainer 115, and W n is the associated output. m is Y m ={X1,X2,.....X g}, and Z m is Z m ={S(X1),S(X2).....S(X g )}. m Typically, g may be 10 to 100 images.

[0096] For each pair [Y m ,Z m ] is the relevant standard image X taken with standard view c at time 0 to n. c Set of W n The time n may indicate a time within the cardiac cycle. As described above, the skilled sonographer 2 may indicate cardiac cycle information and may have a set W n Related standard images that may be included in X c can be provided.

[0097] In this scenario, the periodic standard view trainer 115 receives as input the general frame Y m and their approximate transformation Z generated by the oriented neural network 15. m and their associated cardiac cycle timings n, and trained to generate a set of standard view summary images Wn at the desired timings n. Loss=loss(CC n ,W n ) (10) where CC n is the output of the periodic standard view neural network 110 during training.

[0098] The periodic standard view trainer 115 can use any suitable neural network, such as a fully convolutional network, an encoder / decoder type network, or a generative adversarial network, to generate the trained periodic standard view neural network 110 for the navigator 100.

[0099] As shown in FIG. 9B, the navigator 100''' may include a trained orientation neural network 15, a set creator for operations 112', a sufficiency checker 60', a result converter 40', a trained periodic standard view neural network 110, and a diagnoser 50.

[0100] During operation, the non-sonographer 5 can manipulate the probe 7 near the body part of interest for a period at least long enough to cover a cycle (e.g., a cardiac cycle) of the desired body part. Images from the probe 7 are provided to a trained orientation neural network 15 to generate their associated transformations S(X) and to a set creator 112′ to generate an appropriate set Y m and Z m The sufficiency checker 60' generates the set Y m and Z mis sufficiently large, which may cause the result converter 40' to instruct the user 5 to point the probe 7 in a desired direction or to continue observing in the current orientation. It will be appreciated that in this embodiment, the non-sonographer 5 does not need to hold the probe 7 exactly in the standard view, and thus the instructions provided by the result converter 40' may be coarser. The periodic standard view neural network 110 derives the summarized periodic standard view CC from the output of the set creator 112'. n can be generated.

[0101] It will be appreciated that this embodiment is also useful for non-periodic body parts, particularly where the user 5 may be holding the probe 7 unsteadily. In this embodiment, each set may only have one or two images in it.

[0102] Applicant has further realized that a neural network can be trained without the transformation information generated by a trained directional neural network 15. This is shown in Figures 10A and 10B, which show a system similar to Figures 9A and 9B, but without a trained directional neural network 15. As a result, for training (Figure 10A), the set creator 113 generates a set of images X i From Y m At time n, a standard image X c From W n The Periodic Standard View Trainer 115 can generate the Periodic Standard View Neural Network 110 using Equation (10).

[0103] At run time (FIG. 10B), the set creator 113' creates an image X i From Y m The periodic standard view neural network 110 creates a summary view CC n can be generated.

[0104] It will be appreciated that the present invention can provide a navigator for non-sonographers to operate a mobile ultrasound device without training and without the use of additional hardware other than an ultrasound probe. Thus, the navigator of the present invention receives ultrasound images as its only input. It will be further appreciated that this allows non-sonographers to perform ultrasound scans in many non-traditional scenarios, such as ambulances, battlefields, emergency care facilities, nursing homes, etc.

[0105] Additionally, the present invention may be implemented in more conventional scenarios, such as part of a conventional device used in a hospital or clinic environment, or it may be implemented on a cart.

[0106] Unless otherwise indicated, as is evident from the above discussion, discussions throughout this specification using terms such as “processing,” “computing,” “calculating,” “determining,” and the like, refer to the operations and / or processes of any type of general-purpose computer, and it should be understood that such computers include client / server systems, mobile computing devices, smart appliances, or similar electronic computing devices that manipulate and / or transform data represented as physical quantities, e.g., electronic quantities, in the registers and / or memory of the computing system into other data represented as physical quantities in the memory, registers, or other information storage, transmission, or display devices of the computing system.

[0107] Embodiments of the present invention may include an apparatus for performing the operations described herein. This apparatus may be specially configured for the desired purposes, or may comprise a general-purpose computer or client / server configuration selectively activated or reconfigured by a computer program stored on the computer. The resulting apparatus, when executed by software, can transform a general-purpose computer into an element of the present invention as described herein. The executable instructions can define a device of the present invention operating on a desired computer platform. Such a computer program may be stored on a computer-accessible storage medium, which may be a non-transitory medium, including, but not limited to, an optical disk, a magneto-optical disk, read-only memory (ROM), volatile and non-volatile memory, random access memory (RAM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, flash memory, a disk-on-key, or any other type of medium suitable for storing electronic instructions and capable of coupling to a computer system bus.

[0108] The processes and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the desired method. The desired structure for a variety of these systems will appear from the description below. Further, embodiments of the present invention are not described with reference to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the present invention as described herein.

[0109] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.

Claims

1. 1. An apparatus for an ultrasound unit having an ultrasound probe, comprising: a processor, the processor comprising: a trained oriented neural network that receives as input at least one non-standard image of a body part from the ultrasound unit and generates as output a non-discrete matrix transformation that converts between the position and rotation in three-dimensional space associated with the standard image and the position and rotation in three-dimensional space associated with the non-standard image; a result converter that converts the transformation into an indication of a position and / or rotation of the probe and displays the indication of the position and / or rotation to a user to modify the position and / or rotation of the probe.

2. 10. The apparatus of claim 1, An apparatus comprising: a trainer for training the orientation neural network, the trainer using the standard image and non-standard images taken around the standard image, and a transformation between a position and / or rotation in space associated with the non-standard image and a position and / or rotation associated with the standard image.

3. 3. The apparatus of claim 2, 10. The apparatus of claim 9, wherein the trainer includes a training converter that receives inertial measurement unit (IMU) data during a training session from an IMU attached to a training probe, the IMU data providing positions and / or rotations associated with the non-standard image and the standard image, and converts the positions and / or rotations into a transformation between the positions and / or rotations associated with the standard image and the positions and / or rotations associated with the non-standard image.

4. 3. The apparatus of claim 2, the trainer includes an untrained oriented neural network and a loss function for training the untrained oriented neural network, the loss function configured to reduce the distance between a computed transformation produced by the untrained oriented neural network and a ground truth transformation of each non-standard image.

5. 10. The apparatus of claim 1, and a diagnoser for making a diagnosis from a final image produced by said probe when in a position and rotation similar to that associated with said standard image.

6. 10. The apparatus of claim 1, a set creator that receives a plurality of images from the probe and a plurality of transformations from the trained oriented neural network and generates a set of pairs, responsive to the plurality of images, each pair having one of the plurality of images and its generated transformations; a sufficiency checker that determines when a sufficient set has been generated; and a trained periodic canonical view neural network for generating a set of periodic canonical images showing changes in the body part during its cycle.

7. 7. The apparatus of claim 6, and a periodic canonical view trainer for training an untrained periodic canonical view neural network using the set of pairs and using associated summary periodic canonical images at each point in the cycle of the body part.

8. 7. The apparatus of claim 6, The apparatus, wherein the body part cycle is a cardiac cycle.

9. 7. The apparatus of claim 6, wherein each of said sets includes a single said pair.

10. 1. A method for an ultrasound unit having an ultrasound probe, the method being implemented on a computing device, comprising: receiving at least one non-standard image of the body part from the ultrasound unit and generating a transformation using a trained oriented neural network that transforms between positions and images associated with the standard image and positions and images associated with the non-standard image; converting the transformation into an indication of a position and / or rotation of the probe and displaying the indication of the position and / or rotation to a user for modifying the position and / or rotation of the probe; prior to said receiving, training said orientation neural network using said standard image and non-standard images taken around said standard image, and transformations between positions and / or rotations in space associated with said non-standard images and positions and / or rotations associated with said standard images; the training step: training an untrained oriented neural network using a loss function; The method, wherein the loss function is configured to reduce the distance between the computed transformation produced by the untrained oriented neural network and a ground truth transformation for each non-standard image.

11. 11. The method of claim 10, the training step includes receiving inertial measurement unit (IMU) data during a training session from an IMU attached to a training probe; The method, wherein the IMU data provides positions and / or rotations associated with the non-standard image and the standard image and translates those positions and / or rotations into a transformation between the positions and / or rotations associated with the standard image and the positions and / or rotations associated with the non-standard image.

12. 11. The method of claim 10, receiving a plurality of images from the probe and a plurality of transformations from the trained oriented neural network, and generating a set of pairs responsive to the plurality of images, each pair having one of the plurality of images and its generated transformations; determining when enough sets have been generated; and generating a set of periodic canonical images using the trained periodic canonical view neural network to show changes in the body part during its cycle.

13. 13. The method of claim 12, training an untrained periodic canonical view neural network using the set of pairs and associated summary periodic canonical images at each point in the cycle of the body part.

14. 13. The method of claim 12, The method, wherein the body part cycle is a cardiac cycle.

15. 13. The method of claim 12, wherein each of said sets contains a single said pair.

Images