Providing ultrasound imaging guidance

Abstract

A mechanism for identifying a recommended path of an ultrasound probe during ultrasound imaging. An image of a surface area covering a desired subsurface area is received (211). The image is processed to identify (212) a recommended path for ultrasound image data of the ultrasound probe that will capture the subsurface region.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of ultrasound imaging, and in particular to providing guidance for an ultrasound imaging procedure. BACKGROUND

[0002] US 2009 / 024030 discloses a method of presenting a suggested path for an ultrasound probe along a surface of a patient, comprising obtaining a three-dimensional non-ultrasound image of the patient and an ultrasound image of the patient. A treatment zone is defined in the three-dimensional non-ultrasound image, and using said image, a scan site profile of a structure of interest in the patient and an external profile of the surface of the patient are defined. A preferred path of an ultrasound scan is projected onto the external profile, such that the operator does not need to know the structure of interest to reproduce the ultrasound scan.

[0003] Ultrasound imaging remains an important non-invasive imaging procedure for assessing the condition of a subject. Ultrasound imaging is generally considered particularly important in the obstetric field, for monitoring the growth and / or condition of a foetus. However, ultrasound imaging also has applications in other fields, in particular cardiac or oesophageal ultrasound imaging. One attractive feature of ultrasound imaging is its portability, since the equipment required to perform ultrasound imaging is relatively compact, and requires less complex equipment and lower power resources than other imaging modalities.

[0004] Indeed, ultrasound imaging has rapidly become an important diagnostic or assessment tool in resource-poor regions of the world. One problem faced by these resource-poor regions is the lack of well-trained sonographers to acquire and interpret ultrasound scan results.

[0005] There is therefore a strong desire to increase and improve the guidance available to a user or operator of an ultrasound imaging probe, to improve ultrasound image capture of a desired subsurface region. Capturing sufficient data is crucial for accurate and reliable interpretation of the condition of a patient.

[0006] There is therefore a need to improve mechanisms for providing ultrasound imaging guidance. SUMMARY

[0007] The invention is defined by the independent claims. The dependent claims define advantageous embodiments.

[0008] In one aspect of the application, there is provided a computer-implemented method for providing ultrasound imaging guidance. The computer-implemented method comprises performing an image display process, the image display process comprising: receiving an image of a surface region along which an ultrasound probe is to be moved during an ultrasound imaging procedure (e.g. from a camera); processing the image of the surface region to identify a recommended path for the ultrasound probe along the surface region, wherein the ultrasound probe will capture a sequence of ultrasound images covering a desired subsurface region if it is moved along the recommended path during the ultrasound imaging procedure; controlling a user interface to display an image; and augmenting the display of the image with a representation of the recommended path superimposed on a representation of the surface region in the displayed image.

[0009] Embodiments thereby provide a mechanism for defining or identifying a recommended path to be taken by an ultrasound probe by processing images of a surface region along which the ultrasound probe is to be moved, is moving or has moved in performing an ultrasound probe imaging procedure. This provides a low cost and adaptive technique for generating ultrasound guidance information which can be used even in resource-poor environments (e.g. environments where processing power and / or guidance equipment is limited).

[0010] In some examples, the method comprises iteratively performing the image display process, wherein the image received in each iteration of the image display process comprises a different image in a sequence of surface region images. In this way, a recommended path can be produced for each frame of a video.

[0011] In some examples, the next image in the sequence of images comprises the most recently available image in a video stream of the surface region. This approach is particularly advantageous for providing continuous guidance throughout the course of an ultrasound imaging procedure, e.g. adjusting the path for the most recent image of the ultrasound imaging procedure to help guide an operator of the ultrasound probe.

[0012] The method can further comprise tracking movement of the ultrasound probe during the ultrasound imaging procedure. The image display process can further comprise: processing the tracked movement of the ultrasound probe to identify an actual path taken by the ultrasound probe along the surface region; and augmenting the display of the image with a representation of the actual path superimposed on a representation of the surface region in displaying the image.

[0013] The method provides feedback to the ultrasound probe operator about their successful adherence to the recommended path. For example, this information can be used during the ultrasound imaging procedure to correct the path of the ultrasound probe or repeat missed parts of the recommended path. Alternatively, this information can be used after the ultrasound imaging procedure is completed to teach or train the ultrasound probe operator, e.g. for assessing the performance of the operator. For example, in this way, a measure of the utility or usefulness of any obtained ultrasound data can be defined.

[0014] The method can further comprise tracking movement of the ultrasound probe during the ultrasound imaging procedure. The image display process can further comprise processing the tracked movement of the ultrasound probe to identify an actual path taken by the ultrasound probe along the surface region; determining whether a deviation of the recommended path from the actual path exceeds a predetermined amount; and generating a user-perceptible warning if the recommended path deviates from the actual path by more than the predetermined amount. For example, the signal can comprise any one or a combination of the following: a visual signal, a haptic signal, and an audible signal, if the mismatch exceeds a predetermined threshold. According to this method, for example, the ultrasound probe can vibrate when the deviation between the recommended path and the actual path exceeds a certain threshold.

[0015] The method can further comprise tracking movement of the ultrasound probe during the ultrasound imaging procedure. The image display process can further comprise processing the tracked movement of the ultrasound probe to identify an actual path taken by the ultrasound probe along the surface region; updating the recommended path based on the actual path; and augmenting the display of the image in the display image with a representation of the updated recommended path superimposed on the representation of the surface region. In this way, a minimal amount of disruption to the workflow of the sonographer can be maintained. A smooth workflow, i.e. scanning procedure, is thereby achieved.

[0016] The step of processing the tracked movement of the ultrasound probe can comprise using movement data captured by a movement sensor in the ultrasound probe to track movement of the ultrasound probe.

[0017] In some examples, the method further comprises iteratively receiving ultrasound images captured by the ultrasound probe during the ultrasound imaging procedure; and for each ultrasound image: processing the ultrasound image to determine whether the region of interest is captured in the ultrasound image; and in response to determining that the region of interest is captured, controlling the user interface to provide a user-perceptible warning that the ultrasound probe is capturing ultrasound images of the region of interest.

[0018] Thus, in the method, a check is made to determine whether the anatomical structure of interest (e.g. anatomical feature, element or structure) is detected in a particular frame of the ultrasound imaging process. If a positive determination is made, audio, visual and / or haptic feedback can be sent to the operator (e.g. notification beep, warning box, etc.). This can prompt the operator to leave the probe in that area, for example, in order to acquire more ultrasound image data about the anatomical structure of interest.

[0019] Thus, the method increases the likelihood that useful ultrasound data about a particular region of interest will be obtained. In particular, the method advantageously alerts the operator when the probe is positioned over the region of interest. This in itself provides useful information to the operator (e.g. a clinical decision can be made if the region of interest is in an unusual location, etc.).

[0020] In some examples, the step of processing the image of the surface region comprises processing the image using an ellipsoid fitting algorithm to fit an ellipsoid to the portion of the image representing the surface region; and identifying the path based on at least a size of the fitted ellipsoid.

[0021] In some examples, the step of identifying the path based on at least the size of the fitted ellipsoid comprises using one or more known characteristics of the ultrasound probe.

[0022] In some examples, the one or more known characteristics of the ultrasound probe comprise a maximum distance.

[0023] The maximum distance can be a recommended (maximum) distance between sweep lines, e.g. a distance recommended by the ultrasound probe manufacturer.

[0024] The maximum distance can be a maximum allowed distance between a first position of the ultrasound probe (for imaging a first subsurface region) and a second position of the ultrasound probe (for imaging a second subsurface region), wherein one side of the first subsurface region completely overlaps the second subsurface region.

[0025] In some examples, the step of identifying the path comprises configuring the path such that a displacement between each of at least 80% of all positions along the path and a position at at least one distance along the path (where the distance along the path is greater than the maximum distance) is less than or equal to the maximum distance.

[0026] The method increases the likelihood that moving the ultrasound probe along the path captures sufficient ultrasound data of the subsurface region of interest to image the subsurface region completely, e.g. to a sufficient level of detail to facilitate accurate reconstruction of the subsurface region in ultrasound image space.

[0027] The surface region can be part of the abdominal surface or pelvic surface of an individual. For example, the subsurface region can be the uterus.

[0028] The recommended path can take the shape of a grid. This provides an easy to follow and repeatable path for the ultrasound probe to follow for unskilled or inexperienced operators to follow.

[0029] A computer program product comprising computer program code units which, when executed on a computing device having a processing system, cause the processing system to perform all the steps of any of the methods disclosed herein is also presented.

[0030] Furthermore, a user interface system for providing ultrasound imaging guidance is also presented, the user interface system comprising a user interface and a processing system communicatively coupled with the user interface. The processing system is configured to perform the functions implemented by any of the methods disclosed herein. The user interface or the processing system can comprise or be provided with an image capturing device (e.g. a camera).

[0031] These and other aspects of the application will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0032] For a better understanding of the present application, and to show how it can be put into effect, there will now be described by way of example only, reference being made to the accompanying drawings in which:

[0033] Figure 1 Figures illustrate use cases for the presented embodiments;

[0034] Figure 2 is a flowchart illustrating the presented method;

[0035] Figure 3 the flowchart of illustrates a technique for determining an ultrasound probe path;

[0036] Figure 4 is a flowchart illustrating the presented method;

[0037] Figure 5 is a flowchart illustrating the presented method;

[0038] Figure 6 is a flowchart illustrating the presented method; and

[0039] Figure 7 illustrates a user interface system;

[0040] Figure 8 illustrates a path schematic according to an embodiment. DETAILED DESCRIPTION

[0041] The application will be described with reference to the Figures.

[0042] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of apparatuses, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the present application. These and other features, aspects, and advantages of the apparatuses, systems and methods of the present application will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the use of the same reference numerals in different Figures is intended to represent the same or similar

[0043] The present application provides a mechanism to identify a recommended path for an ultrasound probe in an ultrasound imaging procedure. An image of a surface region covering a desired subsurface region is received. The image is processed to identify a recommended path for the ultrasound probe that will capture ultrasound image data of the subsurface region.

[0044] Embodiments of the present application are based on the recognition that it can be possible to predict a path that an ultrasound probe should travel in order to capture ultrasound data of a particular region of interest.

[0045] The proposed method can be used in any environment where a medical subject is being imaged using ultrasound, but is particularly advantageous in environments where resources are scarce and / or the level of expertise of the operator of the ultrasound probe is low.

[0046] Figure 1 Scenarios in which the proposed embodiments can be used are schematically illustrated in order to improve the understanding of the context. In particular, Figure 1 A situation is illustrated in which a subject 190 is to undergo, is undergoing or has undergone ultrasound imaging.

[0047] It is known that, in order to perform ultrasound imaging, an operator moves an ultrasound probe 110 over a surface region 191 of the subject 190 (e.g. the skin covering one or more internal organs). The ultrasound probe iteratively generates ultrasound data, e.g. in response to echoes of ultrasound emitted by the ultrasound probe. For example, the ultrasound transducer 111 can emit ultrasound waves and, in accordance with known procedures, generate ultrasound data in response to any echoes responsive to the emitted ultrasound. The ultrasound data can comprise ultrasound image data (e.g. one or more 2D or 3D ultrasound images), or any suitable data from which a sequence of one or more 2D or 3D ultrasound images can be derived. For example, the ultrasound probe can provide raw ultrasound data to an external ultrasound processing system (not shown) which processes the raw ultrasound data to produce ultrasound image data. The ultrasound (image) data represents a subsurface region 192 of the individual, e.g. containing information about the part of the individual covered by the surface region 191.

[0048] By way of working example only, the surface region can be a part of the abdominal / pelvic surface of a human body. The subsurface region can be or include the uterus, for example, for performing a fetal ultrasound imaging procedure. Other suitable examples of surface regions and corresponding subsurface regions will be apparent to the skilled person and will depend on the exact use case scenario. For example, in one example use case scenario, the surface region is the surface of the chest and the subsurface region is then the chest or thorax (e.g. including at least the heart and / or lungs). As another example, the surface region is the surface of the pelvis and the subsurface region is then the reproductive organs (such as the uterus). As yet another example, the surface region is the surface of the back and the subsurface region includes the soft tissue of the head and / or neck, for example including at least the lymph nodes.

[0049] It has been recognized that, in order to capture ultrasound (image) data of the desired subsurface region 192, for example a series of ultrasound images, the ultrasound probe must be moved accurately over the surface region 191 so that the ultrasound probe is correctly positioned to capture all parts of the desired subsurface region.

[0050] The present disclosure discloses a mechanism for determining a path to be taken by the ultrasound probe 110 (along the surface region) in order to capture a particular desired subsurface region. This determined path is then used to enhance an image and / or video of the surface region.

[0051] This enhanced image and / or video can be provided prior to the ultrasound imaging procedure, for example, to provide a recommended path for performing the ultrasound imaging procedure. In another approach, the enhanced image and / or video is provided during the ultrasound imaging procedure, for example, to provide active feedback to the operator of the ultrasound probe. In yet another approach, the enhanced image and / or video can be provided after the ultrasound imaging procedure is completed, for example, as a teaching aid to help guide the operator in possible improvements in the movement of the ultrasound probe.

[0052] Figure 1 It is also illustrated how the ultrasound probe 110 can comprise a motion sensor 112 for tracking or monitoring the movement of the ultrasound probe. For example, such a motion sensor can comprise an accelerometer or a gyroscope. Other types of motion sensors are known in the art.

[0053] In the context of the present disclosure, a "path" refers to a route or trajectory, or a combination of routes or trajectories, taken by the ultrasound probe while in contact with the surface region. Thus, the path can be continuous (e.g. serpentine) or discontinuous (e.g. a grid).

[0054] Figure 2A flowchart illustrates a computer-implemented method 200 for providing ultrasound imaging guidance. The computer-implemented method comprises performing an image display process 210, which can be iteratively performed.

[0055] The image display process 210 comprises a step 211 of receiving or obtaining an image of a surface region along which an ultrasound probe is to be moved during an ultrasound imaging procedure. The image is initially generated by a camera or other image capture device located externally of the subject, and can be received from the image capture device and / or memory. The image of the surface region is thus an external image of the subject of the ultrasound imaging being performed (e.g. to be performed or previously performed).

[0056] The image display process 210 further comprises a step 212 of processing the image of the surface region to identify a recommended path for the ultrasound probe along the surface region. The recommended path is configured such that, if the ultrasound probe is moved along the recommended path during the ultrasound imaging procedure, the ultrasound probe will capture ultrasound data covering a desired subsurface region.

[0057] In some examples, the recommended path takes the shape of a grid. This provides a path that is easy to follow or replicate for inexperienced operators. However, this shape is not essential, and the recommended path can take other shapes (e.g. a serpentine or a set of parallel lines).

[0058] The image display process 210 further comprises a step 213 of controlling a user interface to display the image. Methods for controlling a user interface to display an image are well known in the art, and are not described in detail for the sake of brevity.

[0059] The image display process 210 further comprises a step 214 of augmenting the display of the image with a representation of the recommended path, which is superimposed over the representation of the surface region in the displayed image. Methods for augmenting an image with additional information (e.g. annotating the image) are also well known. The spatial relationship between parts of the image and the recommended path is known when the image is processed to identify the path. This facilitates easy augmentation of the image with the recommended path.

[0060] As mentioned previously, the image display process can be iteratively repeated.

[0061] In preferred embodiments, for each iteration of the image display process, the image received in each iteration comprises a different image of a sequence of images of the surface region. For example, the sequence of images can define a video of the surface region, and the different image can be the next image in the sequence of images.

[0062] In preferred examples, the image received by any iteration of the image display process is the most recently available image of the video stream of the surface region. This is effective to facilitate the provision of a dynamically updated video stream which is augmented to provide a representation of the determined path. This approach is particularly advantageous if it is performed during an ultrasound imaging procedure, for example to provide immediate and ongoing feedback to an operator of the ultrasound probe.

[0063] However, in some examples, the method (or at least steps 213 and / or 214) can also / or can be performed before and / or after the ultrasound imaging procedure. Performing this before the ultrasound imaging procedure facilitates guiding the ultrasound imaging procedure in advance. Performing the method after the ultrasound imaging procedure facilitates providing feedback to the operator.

[0064] Thus, in some examples, at least step 214 (and optionally step 213) is performed (only) before the ultrasound imaging procedure (representing an "initial mode"). In other examples, at least step 214 (and optionally step 213) is performed (only) during the ultrasound imaging procedure (representing a "live mode"). In yet other examples, at least step 214 (and optionally step 213) is performed (only) after the ultrasound imaging procedure (representing a "teaching mode").

[0065] In the method of iteratively repeating the image display process, step 212 can comprise adjusting or updating the path based on the path generated for a previous iteration of the image display process. For example, this can comprise identifying a spatial displacement between the image received in the current iteration of the image display process and the image received in the iteration of the previous image display process. The path can then be subjected to a corresponding spatial displacement. Methods for determining spatial displacements are well known in the art of image processing.

[0066] Figure 3 The flowchart of Figure 3 illustrates a method of performing step 212 to identify a path along the surface region.

[0067] The proposed step 212 comprises a sub-step 310 of processing the image using an ellipsoid fitting algorithm to fit an ellipsoid to the portion of the image representing the surface region.

[0068] For example, sub-step 310 can be performed by first identifying the portion of the image representing the surface region. This can be performed using any suitably trained segmentation algorithm, for example a machine learning algorithm and / or a deformable contour algorithm.

[0069] A variety of segmentation algorithms are known in the art, for example those mentioned in the following articles: Minaee, Shervin, et al. "Image segmentation using deep learning: A survey." IEEE transactions on pattern analysis and machine intelligence 44.7 (2021): 3523-3542; He, Lei, et al. "A comparative study of deformable contour methods on medical image segmentation." Image and vision computing 26.2 (2008): 141-163; or Ghosh, Swarnendu, et al. "Understanding deep learning techniques for image segmentation." ACM Computing Surveys (CSUR) 52.4 (2019): 1-35. Any one or more of the identified segmentation techniques can be used and / or adapted for performing step 311, and other segmentation techniques will be apparent to the skilled person.

[0070] Step 312 can then perform an ellipsoid fitting step 312 using an ellipsoid fitting algorithm to fit an ellipsoid over the identified portion. Of course, step 312 can be omitted if, for example, the segmentation algorithm is configured to directly produce an ellipsoid that identifies the portion of the image representation surface area.

[0071] For example, step 312 can be performed using the OpenCV fitEllipse function to fit an ellipse to the identified shape and position of the portion. Another approach is to employ a direct least squares fitting of an ellipse method, taking the edges of the identified portion in the image as data points to which the ellipse is to be fitted. Other ellipsoid fitting techniques will be apparent to the skilled person.

[0072] In another example, sub-step 310 can be performed using a geodesic fitting technique that leverages one or more single image depth estimation techniques that have been developed. An example of a suitable technique is presented by Mertan, Alican, Damien Jade Duff, and Gozde Unal in "Single image depth estimation: An overview." Digital Signal Processing (2022): 103441.

[0073] As a working example, the ellipsoid equation can be defined in the following form:

[0074] Every point on the ellipsoid should satisfy this equation, or (from an estimation paradigm perspective), the absolute error of the actual points in this equation should be minimized with respect to the best estimate of the parameters. Depth estimation can be used to estimate the parameters of this equation, assuming that the z-axis origin roughly corresponds to the front-most region of the image.

[0075] The proposed step 212 also includes sub-step 320, which is to identify a path based at least on the size of the fitted ellipsoid. For example, based on the area of the detected surface region (e.g., represented by the fitted ellipsoid), the number and / or location of one or more sweep lines to be taken by the ultrasound probe can be identified. For example, there can be a predetermined mapping between the size of the fitted ellipsoid and the number of (recommended) sweep lines.

[0076] In some examples, step 320 includes using the depth information, along with the fitted ellipsoid, to identify the size of the surface region. Specifically, the depth information can be used to define the distance between different points in the image of the surface region. This facilitates the real identification of the size of the surface region (e.g., the size of the fitted ellipsoid). The identified size of the surface region can be used to define or determine a path to be taken by the ultrasound probe to capture ultrasound data of the desired sub-region. For example, the depth information can be estimated using any suitable depth information technique, such as presented by Mertan, Alican, Damien Jade Duff, and Gozde Unal in "Single image depth estimation: An overview." Digital Signal Processing (2022): 103441.

[0077] There can be a predetermined mapping between the size of the fitted ellipsoid (whether a relative size in the image or a determined corresponding actual size) and the number of (recommended) sweep lines and / or a recommended sweep line pattern. A sweep line refers to the movement of the ultrasound probe along a single curve or straight line, which is known in the art.

[0078] In an example, information about the ultrasound probe (and / or the ultrasound system using the ultrasound probe) is used to calculate or determine the path. Thus, step 320 can comprise determining the path using one or more known characteristics of the ultrasound probe. Thus, step 212 can comprise a sub-step 330 of obtaining one or more (known) characteristics of the ultrasound probe.

[0079] As a working example, the one or more (known) characteristics of the ultrasound probe comprise a recommended distance or a maximum distance. For example, the recommended / maximum distance can define a maximum recommended distance between two sweep lines of the ultrasound probe (e.g. a distance recommended by the manufacturer). For example, the maximum distance can define a maximum allowed distance between two positions of the ultrasound probe, which respectively image a fully overlapping side of a subsurface region. In this way, the maximum distance can define a maximum allowed distance between a first position of the ultrasound probe (for imaging a first subsurface region) and a second position of the ultrasound probe (for imaging a second subsurface region), wherein a side of the first subsurface region fully overlaps the second subsurface region. For example, a right edge of an ultrasound image covering the first subsurface region can fully overlap a left edge of a second ultrasound image covering the second subsurface region, thereby avoiding any imaging gap between the first image and the second image. In other examples, the “side” can be a side of the image, e.g. a left half, a right half, an upper half or a lower half, and the right side of the first image overlaps the left side of the second image. This overlap can be full or partial.

[0080] Figure 8 Fig. 8 illustrates a path 804 that the probe will take when imaging a first subsurface region 801 and a second subsurface region 802. The path 804 is configured such that the probe will take a position 811 when imaging the first subsurface region 801 and a position 812 when imaging the second subsurface region 802. The path 804 is configured such that the probe will take a position 813 when imaging a third subsurface region 803 and a position 814 when imaging a fourth subsurface region 804.

[0081] By selecting a path that uses the recommended distance or the maximum distance, it is facilitated to determine a path that captures sufficient ultrasound data to image a large portion of the entire desired subsurface region.

[0082] For example, the step 320 of identifying the path can comprise configuring the path according to Figure 8 such that, between each position of at least 80% of all positions 811, 812, 813, 814 along the path 804 and at least one position 814 of the distance 824 along the path 804 (where the distance 824 along the path is greater than the maximum distance), the displacement (i.e. the shortest distance between two points) is less than or equal to the maximum distance. In other words, with reference to Fig. 8, the path 804 is configured such that, between each position of at least 80% of all positions 811, 812, 813, 814 along the path 804 and at least one position 814 of the distance 824 along the path 804 (where the distance 824 along the path is greater than the maximum distance 822), the displacement (i.e. the shortest distance between two points) is less than or equal to the maximum distance. In other words, with reference to Fig. 8, the path 804 is configured such that, between each position of at least 80% of all positions 811, 812, 813, 814 along the path 804 and at least one position 814 of the distance 824 along the path 804 (where the distance 824 along the path is greater than the maximum distance 822), the displacement (i.e. the shortest distance between two points) is less than or equal to the maximum distance. Figure 8During the imaging procedure, the probe can pass along the path 804 through imaging locations 811, 812, 813,... until reaching imaging location 814. The distance travelled by the probe between each imaging location 811 to 812, 812 to 813,... can be less than the maximum distance 822. Thus, the total distance travelled by the probe from location 811 to 814 is the distance 824 along the path 804, which is greater than the maximum distance 822. However, the displacement of the probe relative to 811 at location 814 is less than or equal to the maximum distance 822. This effectively facilitates determining a path that images no less than 80% of the desired subsurface region.

[0083] Figure 4 The flowchart of Figure 4 illustrates a computer-implemented method 400 that provides additional optional steps. The method 400 comprises performing an image display procedure 410, which comprises the steps 211, 212, 213, 214 described above.

[0084] The computer-implemented method 400 further comprises a step 420 of tracking the motion of the ultrasound probe during the ultrasound imaging procedure. Methods for tracking the motion of an ultrasound probe are known and can for example use movement data captured by motion sensors within or on the ultrasound probe to track the movement of the ultrasound probe.

[0085] In a preferred example, the step 420 comprises tracking the motion of the ultrasound probe with respect to the image received in step 211. This can be achieved for example by processing the image received in step 211 to identify the relative positioning of the image representation of the ultrasound probe in the image. In this way, it is possible to directly produce the tracked (actual) path in the same reference frame as the recommended path.

[0086] In other words, any surface region image obtained during the ultrasound imaging procedure will contain a representation of the ultrasound probe. It is therefore possible to track the positioning of the ultrasound probe using the same camera that is used to determine the recommended path. Methods for tracking the ultrasound probe through a sequence of images can make use of one or more optical flow algorithms (as the magnitude of the probe’s movement will be significantly more than other parts of the image) and / or object recognition algorithms (e.g. segmentation algorithms designed to identify the positioning of the ultrasound probe).

[0087] In another example, the movement of the ultrasound probe can be tracked in a separate coordinate frame / space (i.e. a different reference frame). Techniques for tracking objects in three-dimensional space are known, for example accelerometer-based tracking, electromagnetic tracking, infrared (light) tracking and mechanical positioning systems. US patent applications US2021 / 186622A1; US2013 / 245428A1 and US2010 / 268072A1 disclose methods of tracking the motion of an ultrasound probe (and thus facilitating tracking of its path).

[0088] The image display process 410 can further comprise a step 411 of processing the tracked motion of the ultrasound probe to identify an actual path taken by the ultrasound probe along the surface region. For example, this can be performed by iteratively recording the movement to track the path or route taken by the ultrasound probe along the surface.

[0089] The actual path can be defined relative to the image and / or three-dimensional space.

[0090] The image display process 410 can further comprise a step 412 of augmenting the display of the image by superimposing a representation of the actual path on the representation of the surface region in the displayed image.

[0091] If the actual path is defined in three-dimensional space (e.g. rather than directly relative to the displayed image), then the spatial relationship between the displayed image and the actual path can be readily determined using standard procedures.

[0092] For example, the coordinate system for the tracked (real) path can be mapped or spatially registered to the reference frame of the displayed image. This facilitates spatial registration of the real path with the displayed image.

[0093] For example, the (displayed) image can be spatially registered to three-dimensional space (to track the actual path of the probe) using a calibration procedure. For example, the probe can be positioned at a plurality of predetermined locations of the surface region. For each of these predetermined locations, the position of the probe in three-dimensional space is recorded. Likewise, for each of these predetermined locations, the location in the image representing the predetermined location (e.g. as identified by a segmentation procedure) is recorded. For each predetermined location, the location in three-dimensional space is related to the location in the image. The image is then mapped or warped to three-dimensional space. This mapping can make use of distance information, for example distance information derived from predicted depth information for the image. In this way, the relative spatial registration relationship between the position in three-dimensional space and the corresponding position represented in the image can be determined.

[0094] One example exploits the assumption that the present invention is used to process a series of images (e.g. a video stream) of an object region of interest for which camera motion is negligible. The sequence will start with frames that have no representation of the probe (or, if there is a probe, it will be part of a background that is otherwise stationary (apart from the apparent motion due to slight movement of the camera). Motion estimation methods such as optical flow will be able to locate the probe with reasonable accuracy. If the variation in camera position in the fixed background parts is negligible, then known image registration methods can determine the shift in coordinates quite well. Thus, the net displacement of the probe can be estimated or roughly derived (e.g. we have the displacement of the background in the frame of interest from the reference state via image registration, and the displacement of the probe relative to the previous frame via the optical flow method).

[0095] In examples, an image-based detector can be used. The image-based detector can detect the probe in the image as well as other key markers or predetermined positions of the surface region (e.g. navel and / or other markers of the abdomen, etc.), resulting in the relative position of the probe with respect to each marker. This can be used to relax the restrictions on small camera movements, while mapping these marker coordinates to special values, e.g. the origin, etc.

[0096] The first two approaches can make the measurement somewhat dependent on the current frame.

[0097] Enhancing the image in this way can allow for identifying and / or comparing the recommended path for the ultrasound probe with the actual path. This can be useful, for example, for providing direct feedback during the ultrasound imaging procedure or for providing post-procedure feedback, e.g. for automatically providing feedback to the ultrasound probe operator about their performance.

[0098] In some examples, at least step 412 (and optionally, step 214, and optionally, step 213) is performed only before the ultrasound imaging procedure (denoting an "initial mode"). In other examples, at least step 412 (and optionally, step 214, and optionally, step 213) is performed only during the ultrasound imaging procedure (denoting a "real-time mode"). In yet other examples, at least step 412 (and optionally, step 214, and optionally, step 213) is performed after the ultrasound imaging procedure (denoting a "teaching mode").

[0099] Figure 5 The flowchart illustrates a variant of the computer-implemented method 500 described with reference to Figure 4 The steps involving the same features are identified using the same reference numerals. The method 500 is designed to be performed during the ultrasound imaging procedure, as it can provide direct feedback to the operator of the ultrasound probe.

[0100] In this method, a determination step 510 is performed to determine whether the difference between the recommended path and the actual path exceeds a predetermined amount, e.g. whether the average distance between the actual path and the recommended path is greater than a certain predetermined value, or whether the distance between any point on the actual path and the recommended path is greater than a predetermined value.

[0101] Step 510 can require spatial registration of the actual path and the recommended path (or vice versa) to facilitate determining the distance between the two paths. Spatial registration can be required, for example, if the two paths are not initially defined in the same coordinate system, e.g. with respect to the image used to derive the recommended path.

[0102] Since the recommended path can be defined relative to the image and the actual path can be defined relative to a specific three-dimensional space, e.g. if tracking is performed in three-dimensional space, the actual path and the recommended path can be registered together based on the previously described method of registering the image to the three-dimensional space.

[0103] If a positive determination is made in step 510 (i.e. the difference between the recommended path and the actual path exceeds the predetermined amount), a user-perceptible warning can be generated in step 520. The user-perceptible warning comprises any suitable sensory output (e.g. visual, audible and / or tactile output) that can be perceived by an operator (e.g. the ultrasound probe). Examples include an audible warning (e.g. a beeping sound), a visual warning (e.g. a flashing light or a particular colour) or a tactile warning (e.g. a vibration - which can be provided on the ultrasound probe). Otherwise, the method can return to step 411.

[0104] The method allows for providing direct feedback on the accuracy or appropriateness of the path taken by the ultrasound probe during the ultrasound imaging procedure.

[0105] In some embodiments, step 412 can be omitted. Thus, in some embodiments of the computer-implemented method 500, step 412 is optional.

[0106] Alternatively, the recommended path can be updated based on the actual path. For example, if a continuous deviation of the actual path from the recommended path is identified, the recommended path can be shifted to match the actual path. Since a correct imaging of certain regions can not be possible based on just shifting the recommended path, a new recommended path line can also be created to image the entire region of interest. The updating of the recommended path is performed in a substantially similar manner to the identification of the recommended path. That is, by processing the image of the surface region, wherein at least part of the actual path is used as a boundary or a starting condition. By using this method, the user is enabled to perform a probe sweep more smoothly, even if a deviation between the actual path and the recommended path is found, without the potentially annoying warning and without interrupting the workflow of the sonographer. This way or work can enable a dynamic scan path, which is updated based on the path actually performed.

[0107] Figure 6 The flowchart of Fig. 6 illustrates some further optional features. More specifically, Figure 6 Fig. 6 illustrates a computer-implemented method 600, the method comprising any of the previously described computer-implemented methods 200, 400, 500.

[0108] The computer-implemented method 600 further comprises a step 610 of iteratively receiving ultrasound images from the ultrasound probe during the ultrasound imaging procedure.

[0109] The computer-implemented method 600 further comprises, for each ultrasound image: processing 615 the ultrasound image to determine whether a region of interest (ROI) is captured in the ultrasound image.

[0110] For example, step 615 can be performed using one or more image classification algorithms. One example of such a method is disclosed in European patent application EP3639751 Al. Other examples will be apparent to the skilled person and can employ any suitable classification algorithm to classify the image, e.g. configured to determine whether one or more anatomical / structural elements are present within the image (e.g. an anatomical organ). Example classification algorithms can utilise one or more machine learning algorithms or similar techniques.

[0111] In some examples, the method 600 comprises, in response to determining (in step 615) that a region of interest is captured, controlling 620 a user interface to provide a user-perceptible warning indicating that the ultrasound probe is capturing ultrasound images of the region of interest. Methods of controlling a user interface to perform step 620 are well known in the art. Several example methods have been described previously with reference to similar step 520.

[0112] In some examples, the method 600 comprises, in response to determining (in step 615) that a region of interest is captured, recording 630 the positioning at which the region of interest is captured.

[0113] If the movement of the ultrasound probe is tracked, this can comprise defining or determining the positioning of the ultrasound probe in a coordinate system used to track the ultrasound probe.

[0114] If images of the surface region are captured iteratively throughout the ultrasound imaging procedure, step 630 can comprise recording or saving the image closest to the point in time at which the region of interest is captured in the ultrasound image data. When the ultrasound probe is placed at the surface region, the image of the surface region will contain a representation of the ultrasound probe relative to the surface region. This allows the relative positioning of the ultrasound probe to be identified.

[0115] In some examples, the method 600 further comprises, after the scan is completed, augmenting the display of the surface region with the identified positioning(s). For example, this can be performed by mapping the recorded positioning to a positioning on the displayed image, e.g. using the techniques described previously.

[0116] Steps 630 and 640 facilitate identifying locations from which ultrasound images of interest were obtained. This can provide valuable information to the clinician. For example, the locations can identify areas that can benefit from further ultrasound imaging, e.g., to obtain more information. As another example, the location information can identify anatomical characteristics, e.g., the location of a particular anatomical structure or the orientation of an anatomical structure (such as a fetus). By way of example, if the areas of interest include the head of a fetus and the heart / hips of the fetus, then displaying the locations at which ultrasound images capturing these areas of interest were obtained would facilitate identifying the orientation of the fetus.

[0117] Figure 7 A user interface system 700 for providing ultrasound imaging guidance is illustrated. The user interface system includes a user interface 710 and a processing system 720, which is communicatively coupled to the user interface.

[0118] The processing system 720 is configured to perform image display processing in accordance with any of the methods described herein. In particular, the methods include the steps of: receiving or obtaining an image of a surface region along which an ultrasound probe is to be moved during an ultrasound imaging procedure; processing the image of the surface region to identify a path for the ultrasound probe along the surface region, wherein the ultrasound probe, if moved along the recommended path during the ultrasound imaging procedure, will capture a sequence of ultrasound images covering a desired subsurface region; controlling the user interface to display the image; and augmenting the display of the image with a representation of the recommended path, which is superimposed on a representation of the surface region in the displayed image.

[0119] The processing system can receive the image(s) of the surface region from a camera 730, which captures images (e.g., visible light images) of the surface region. The camera can be integrated into the same device or product (e.g., a cell phone, tablet, laptop, etc.) that carries the processing system. Alternatively, the camera can be a separate component, e.g., an additional or separate camera. The camera can alternatively be integrated into the user interface 710. The camera 730 can form part of the user interface system 700.

[0120] The processing system 720 can be further configured to communicate with an ultrasound probe 110, e.g., to receive ultrasound image(s) generated by the ultrasound probe or to generate ultrasound images using ultrasound data generated by the ultrasound probe. Thus, the processing system 720 can form part of an ultrasound imaging system. The ultrasound probe 110 can form part of the user interface system 700.

[0121] Although illustrated as separate elements, in some examples the user interface 710 forms part of the same device or product that hosts the processing system, for example the user interface (e.g. screen and optionally speaker) of a smartphone, tablet or laptop computer.

[0122] A person of skill in the art would readily recognize that steps of various above-described methods can be performed by programmed computers or processors. Indeed, various aspects and embodiments of the disclosure can be implemented with various types of processors or processing systems (e.g., general purpose processors, application specific processors, or other processors). Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from another source (e.g. memory or computer-readable storage medium) and execute those instructions in relation to the data to perform various operations required by the individual instructions. A processor can also include one or more internal storage devices and a processor can communicate data to and from the internal storage devices as required. A memory or memories can provide a non-transitory computer-readable storage medium for use by or in connection with an embodiment of the disclosure, a processor or a computer-hybrid system. In the context of this document, a "computer-readable storage medium" can be any media or means that can contain, store, or transport the program for use by or in connection with an embodiment of the disclosure. The computer-readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having a number of wires, a tangible processor (e.g., sunk or mounted on a substrate), a tangible RAM based

[0123] A processing system can be implemented in numerous ways, with software and / or hardware, to perform the various functions required. A processor is one example of a processing system that employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. A processing system can however be implemented with or without employing a processor, and also can be implemented as a combination of software and hardware.

[0124] Examples of processing system components that can be employed in various embodiments of the disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0125] In various implementations, a processor or processing system can be associated with one or more storage media such as volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media can be encoded with one or more programs that, when executed on one or more processors and / or processing systems, perform tasks required by the described methods. Various storage media can be fixed within a processor or processing system or can be transportable, such that the one or more programs stored thereon can be loaded into a processor or processing system.

[0126] It should be understood that the disclosed methods are preferably computer-implemented methods. As such, the concept of a computer program is also put forward, which comprises code (i.e. instructions) for implementing any described method when the program is run on a processing system (e.g. a computer). Thus, different parts, lines or blocks of the computer program according to an embodiment can be executed by a processing system or computer to perform any of the methods described herein.

[0127] A non-transitory storage medium storing or carrying a computer program or computer code that, when executed by a processing system, causes the processing system to perform any of the methods described herein is also put forward.

[0128] In some alternative embodiments, the functions recorded in the block diagram(s) or flowchart(s) can not occur in the order shown in the figure(s). For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in reverse order depending on the functionality involved.

[0129] Variations to the disclosed embodiments can become apparent to those of ordinary skill in the art from a reading of the foregoing disclosure and a study of the drawings. Measures recited in the dependent clauses can be combined with measures recited in the independent clauses in any order.

[0130] In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. If the specification states that something is "adapted to", "configured to", "made to", "arranged to" or "capable of", it means that the thing in question is either adapted, configured, made, arranged or capable to do a certain thing, if the specification states that a device or apparatus "comprises" a part or a component, it means that the device or apparatus comprises the part or component, but it does not exclude other parts or components.

[0131] A single processor or other unit can implement several of the functions recited in the claims. If above it is described that a computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state storage medium supplied together with other hardware, or distributed by other devices, or via a data network, such as the Internet or other wired or wireless telecommunication systems.

[0132] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method (200, 400, 500, 600) for providing ultrasound imaging guidance, the computer-implemented method comprising: receiving (211) images of a surface region (191) along which an ultrasound probe (110) is to be moved during an ultrasound imaging procedure; processing the images of the surface region to identify (212) a recommended path (804) for the ultrasound probe along the surface region, wherein the ultrasound probe will capture a sequence of ultrasound images covering a desired subsurface region (192) if moved along the recommended path during the ultrasound imaging procedure; controlling a user interface to display (213) the images; in the displayed images, augmenting (214) the display of the images with a representation of the recommended path superimposed on a representation of the surface region.

2. The computer-implemented method of claim 1, comprising iteratively performing the method of claim 1, wherein, The images received in each iteration comprise different images in a sequence of the surface region images.

3. The computer-implemented method of claim 2, wherein, The next image in the sequence of images comprises the most recently available image in a video stream of the surface region.

4. The computer-implemented method of any of claims 1 to 3, further comprising tracking (420) movement of the ultrasound probe during the ultrasound imaging procedure, wherein, The image display process (210, 410) further comprises: processing tracked movement of the ultrasound probe to identify (411) an actual path taken by the ultrasound probe along the surface region; and in the displayed images, augmenting (412) the display of the images with a representation of the actual path superimposed on a representation of the surface region; or determining (510) whether a deviation of the recommended path from the actual path exceeds a predetermined amount; and generating (511) a user-perceptible warning if the deviation of the recommended path from the actual path exceeds the predetermined amount, wherein the user-perceptible warning preferably comprises any one or a combination of the following: a visual output, a haptic output, and an audible output; or updating the recommended path based on the actual path; and in the displayed images, augmenting the display of the images with a representation of the updated recommended path superimposed on a representation of the surface region.

5. The computer-implemented method of any one of claims 1 to 4, wherein, The step of processing tracked movement of the ultrasound probe comprises using movement data captured by a movement sensor (112) in the ultrasound probe to track movement of the ultrasound probe.

6. The computer-implemented method of any one of claims 1 to 5, further comprising: iteratively receiving (610) ultrasound images captured by the ultrasound probe during the ultrasound imaging procedure; and for each ultrasound image: processing (615) the ultrasound image to determine whether a region of interest is captured in the ultrasound image; and in response to determining that a region of interest is captured, controlling the user interface to provide (620) a user-perceptible warning that the ultrasound probe is capturing ultrasound images of a region of interest.

7. The computer-implemented method of any one of claims 1 to 6, wherein, The step of processing the images of the surface region comprises: processing (310) the image using an ellipsoid fitting algorithm to fit an ellipsoid to the portion of the image representing the surface region; and identifying (320) the recommended path based at least on a size of the fitted ellipsoid.

8. The computer-implemented method of claim 7, wherein, The step of identifying the recommended path based at least on the size of the fitted ellipsoid includes using one or more known characteristics of the ultrasound probe.

9. The computer-implemented method of claim 8, wherein, The one or more known characteristics of the ultrasound probe include a maximum distance (822) that is a maximum allowed distance between a first position (811) of the ultrasound probe for imaging a first subsurface region and a second position (812) of the ultrasound probe for imaging a second subsurface region, wherein one side of the first subsurface region completely overlaps the second subsurface region.

10. The computer-implemented method of claim 9, wherein, The step of identifying the recommended path includes configuring the recommended path such that a displacement (826) between: each of at least 80% of the positions (811, 812, 813, 814) along the recommended path (804), and at least one position (814) at a distance (824) along the recommended path (804) that is greater than the maximum distance (822).

11. The computer-implemented method of any one of claims 1 to 10, wherein, The surface region is a portion of an abdominal surface or a pelvic surface of an individual.

12. The computer-implemented method of any one of claims 1 to 11, wherein, The subsurface region is a uterus.

13. The computer-implemented method of any one of claims 1 to 12, wherein, The recommended path takes the shape of a grid.

14. A computer program product comprising computer program code which, when operating on a computing device having a processing system (720), causes the processing system to perform the method of any one of claims 1 to 13.

15. A user interface system (700) for providing ultrasound imaging guidance, the user interface system comprising a user interface (710) and a processing system (720) communicatively coupled with the user interface, the processing system configured to perform the method of any one of claims 1 to 13.

Images