Mark-up apparatus and method for use with OEM ultrasound scanner

Abstract

The mark-up apparatus includes a fiducial marking arrangement for being manually securely placed on a cable strain relief for visually distinctively marking the cable strain relief, a stereo camera for acquiring a live stereo video of an ultrasound scan probe scanning a patient's region of interest and a mark-up module for processing the live stereo video of the ultrasound scan probe scanning the patient's region of interest and overlaying navigation pointer mark-up(s) to the marked-up live video for instructing a human operator how to manipulate the ultrasound scan probe relative to the patient's region of interest.

Description

[0001] MARK-UP APPARATUS AND METHOD FOR USE WITH OEM UETRASOUND SCANNER

[0002] Field of the Invention

[0003] This invention relates to the field of medical devices in general and apparatus and method for assisting ultrasound scanning in particular.

[0004] Background of the Invention

[0005] Original Equipment Manufacturers (OEMs) often provide so-called closed scan systems including an operating system without an Application Programming Interface (API) thereby preventing external access to the operating system, for example, for augmenting a live scan of a patient’s region of interest.

[0006] There is a need for apparatus and method for use with OEM scan systems in general and OEM ultrasound scanners in particular for assisting ultrasound scanning.

[0007] Summary of the Invention

[0008] The present invention is directed towards mark-up apparatus and method for use with an OEM ultrasound scanner. The OEM ultrasound scanner includes an ultrasound scan probe with a longitudinal ultrasound scan probe centerline for placing on a patient’s region of interest for acquiring a live ultrasound scan, an ultrasound scan controller having an ultrasound image display for displaying a live ultrasound image of the live ultrasound scan, and an ultrasound scan cable for connecting the ultrasound scan probe and the ultrasound scan controller, the ultrasound scan cable having an elongated cable strain relief at its connection to the ultrasound scan probe, the cable strain relief having a longitudinal cable strain relief centerline highly co-directional with the ultrasound scan probe centerline. The ultrasound scan probe can have an integral ultrasound scan cable with an elongated cable strain relief. Alternatively, the OEM ultrasound scanner can include a discrete ultrasound scan cable with an elongated cable strain relief for connection to an ultrasound scan probe.

[0009] The mark-up apparatus includes a combination of hardware components and software components as follows: A fiducial marking arrangement for being manually securely placed on a cable strain relief for visually distinctively marking the cable strain relief. A stereo camera for acquiring a live stereo video of an ultrasound scan probe scanning a patient’s region of interest. A mark-up module for performing the steps of i) processing the live stereo video of the ultrasound scan probe scanning the patient’s region of interest for tracking the fiducial marking arrangement for determining 3D orientation of the ultrasound scan probe centerline with respect to a 3D coordinate system for overlaying an ultrasound scan probe centerline mark-up to the live video of an ultrasound scan probe scanning the patient’s region of interest to create a marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and ii) overlaying at least one navigation pointer mark-up with respect to the ultrasound scan probe centerline mark-up to the marked-up live video for instructing a human operator how to manipulate the ultrasound scan probe relative to a patient’s region of interest. And a human operator display for displaying the marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and the at least one navigation pointer mark-up. Exemplary navigation pointer mark-ups include inter alia a sliding movement mark-up of an ultrasound scan probe relative to a patient’s region of interest, a rotational movement mark-up of an ultrasound scan probe relative to a patient’s region of interest, a tilt movement mark-up of an ultrasound scan probe relative to a patient’s region of interest, an upright movement mark-up of an ultrasound scan probe towards or away from a patient’s region of interest, and the like. The present invention supplements OEM ultrasound scanners and is OEM agnostic in the sense that it can be used with a wide range of commercially available OEM ultrasound scanners including an ultrasound scan cable with an elongated cable strain relief for connecting an ultrasound scan controller and an ultrasound scan probe. The present invention can be equally used with presently installed and new OEM ultrasound scanners. The fiducial marking arrangements are preferably implemented as relatively rigid split sleeves for being manually securely placed on a cable strain relief. The split sleeves can be provided in different lengths and / or different diameters for being manually placed on different cable strain reliefs of different OEMs.

[0010] The present invention can be implemented in a range of set-ups from a telemedicine set-up for enabling a human expert to remotely assist a human operator scanning a patient’s region of interest to an Al set-up to assist a human operator scanning a patient’s region of interest. Some set-ups of the present invention require a structured light source for projecting structured light on a patient’s body including his region of interest. The use of structured light is well known in the art for determining 3D structure and topography of objects where there is difficulty in determining correspondence between stereo views because of lack of surface structure and texture required for accurate location of features. The structured light enables the mark-up module to determine an ultrasound scan probe’s 3D position relative to a patient’s body in addition to its 3D orientation with respect to a 3D coordinate system. Determination of an ultrasound scan probe’s position relative to a patient’s body enables real time recording and storing of ultrasound scans with sufficient clinical information for online and / or offline processing and analysis. Such online and / or offline processing and analysis can be employed for machine learning purposes for automatically overlaying navigation pointer mark-ups. An Al set-up can be located at an OEM ultrasound scanner or remote therefrom, or alternatively, can be cloud-based. Brief Description of the Drawings

[0011] In order to understand the invention and to see how it can be carried out in practice, preferred embodiments will now be described, by way of non-limiting examples only, with reference to the accompanying drawings in which similar parts are likewise numbered, and in which:

[0012] Fig. 1 is a pictorial view of a conventional Original Equipment Manufacturer (OEM) ultrasound scanner.

[0013] Fig. 2 is pictorial view an ultrasound scan cable having a cable strain relief at its connection with an ultrasound scan probe.

[0014] Fig. 3 is a schematic view of a tele-medicine set-up of the mark-up apparatus of the present invention for enabling a human expert to remotely assist a human operator scanning a patient’s region of interest.

[0015] Fig. 4 is a pictorial view of a fiducial marking arrangement of the mark-up apparatus of the present invention,

[0016] Fig. 5 is a pictorial view of a flattened fiducial marking arrangement of the mark-up apparatus of the present invention.

[0017] Fig. 6 is a schematic screen shot of a marked-up live video of an ultrasound scan probe scanning a patient’s region of interest with an ultrasound scan probe centerline mark-up.

[0018] Fig. 7 is a schematic screen shot of a marked-up live video of an ultrasound scan probe scanning a patient’s region of interest and a tilt movement mark-up of the ultrasound scan probe relative to the ultrasound scan probe mark-up.

[0019] Fig. 8 is a schematic screen shot of a marked-up live video of an ultrasound scan probe scanning a patient’s region of interest and a down movement mark-up of the ultrasound scan probe relative to the ultrasound scan probe centerline markup.

[0020] Fig. 9 is a schematic screen shot of a marked-up live video of an ultrasound scan probe scanning a patient’s region of interest and a clockwise rotation movement mark-up of the ultrasound scan probe relative to the ultrasound scan probe centerline mark-up.

[0021] Fig. 10 is a schematic view of a tele-medicine set-up of mark-up apparatus of the present invention with recording / storing functionality.

[0022] Fig. 11 is a schematic view of an Al set-up of mark-up apparatus of the present invention.

[0023] Fig. 12 is a flow diagram of a mark-up method of the present invention.

[0024] Detailed Description of the Drawings

[0025] The present description includes the following sections:

[0026] Section 1 : Original Equipment Manufacturer (OEM) ultrasound scanner

[0027] Section 2: Tele-medicine set-up of mark-up apparatus

[0028] Section 3: Tele-medicine set-up of mark-up apparatus with recording / storing functionality

[0029] Section 4: Al set-up of mark-up apparatus

[0030] Section 5 : Mark up method of the present invention

[0031] Section 1 : Original Equipment Manufacturer (OEM) ultrasound scanner

[0032] Figure 1 and Figure 2 show a conventional OEM ultrasound scanner 100 including an ultrasound scan probe 101 having a longitudinal ultrasound scan probe centerline 102. The ultrasound scan probe 101 is intended to be placed on a patient’s region of interest for acquiring a live ultrasound scan. The OEM ultrasound scanner 100 includes an ultrasound scan controller 103 having an ultrasound image display 104 for displaying a live ultrasound image of a live ultrasound scan. The OEM ultrasound scanner 100 includes a discrete ultrasound scan cable 106 for connecting the ultrasound scan probe 101 and the ultrasound scan controller 103. Alternatively, the ultrasound scan probe 101 can be provided with an integral ultrasound scan cable 106. The ultrasound scan probe 101 can be handheld by a human operator or mounted on an articulated arm for assisting a human operator to place the ultrasound scan probe 101 on a patient’s region of interest for scanning thereat.

[0033] The discrete or integral ultrasound scan cable 106 has an elongated cable strain relief 107 at its connection to the ultrasound scan probe 101. The cable strain relief 107 is designed to be relatively rigid compared to the ultrasound scan cable 106. Typical dimensions of a cable strain relief 107 are about 5 cm to 7 cm long and 1 cm diameter. Accordingly, the cable strain relief 107 has a longitudinal cable strain relief centerline 108 which is highly co-directional with the ultrasound scan probe centerline 102.

[0034] Section 2: Tele-medicine set-up of mark-up apparatus

[0035] Figure 3 shows a tele-medicine set-up 200 A incorporating mark-up apparatus 300 including a mark-up module 310 for live time interfacing therebetween mark-up apparatus components deployed at a human operator location 201 and mark-up apparatus components deployed at a human expert location 202 for enabling a human expert to remotely assist a human operator.

[0036] The human operator location 201 includes the OEM ultrasound scanner 100 and three mark-up apparatus components: A fiducial marking arrangement 320, a stereo camera 330, and a human operator display 340. The human expert location 202 includes three mark-up apparatus components: A first human expert display 350, a human expert interface 360, and a second human expert display 370.

[0037] The fiducial marking arrangement 320 is employed for visually distinctively marking the cable strain relief 107. The fiducial marking arrangement 320 is preferably configured for being manually securely placed on the cable strain relief 107 such that it remains stationary thereon during continuous use of the ultrasound scan probe 101. By virtue of the cable strain relief 107 being relatively rigid, the fiducial marking arrangement 320 is configured to assume a 3D orientation relative to a 3D coordinate system highly co-directional with the cable strain relief centerline 108 and therefore the ultrasound scan probe centerline 102. The 3D coordinate system can be a Cartesian coordinate system or a polar coordinate system.

[0038] The stereo camera 330 has a field of view 331 directed towards a patient. The stereo camera 330 is employed for acquiring a live stereo video of the ultrasound scan probe 101 scanning a patient’s region of interest. The stereo camera 330 is typically stationary and preferably includes Pan Tilt Zoom (PTZ) functionality. The mark-up module 310 processes the stereo camera 330’s live stereo video of the ultrasound scan probe 101 scanning the patient’s region of interest for live tracking of the fiducial marker arrangement 320 for determining 3D orientation of the ultrasound scan probe centerline 102 with respect to a 3D coordinate system. The mark-up module 310 overlays an ultrasound scan probe centerline mark-up to the live video of the ultrasound scan probe 101 scanning a patient’s region of interest.

[0039] The human operator display 340 is employed for displaying a marked-up live video of the ultrasound scan probe 101 scanning a patient’s region of interest including navigation pointer mark-ups with respect to an ultrasound scan probe centerline mark-up for instructing a human operator how to manipulate the ultrasound scan probe 101 relative to a patient’s region of interest. The human operator display 340 is typically adjacent the ultrasound image display 104 for enabling a human operator to readily view both the ultrasound image display 104 and the human operator display 340.

[0040] The first human expert display 350 is in communication with the OEM ultrasound scanner 100 for duplicating the OEM ultrasound scanner’s ultrasound image display 104 in the sense that it displays the same live ultrasound image.

[0041] The human expert interface 360 is employed for enabling a human expert to overlay navigation pointers mark-ups with respect to an ultrasound scan probe centerline mark-up for instructing a human operator how to manipulate an ultrasound scan probe relative to a patient’s region of interest to a live video of an ultrasound scan probe scanning a patient’s region of interest to create a marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and the navigation pointer mark-ups. The human expert interface 360 can be implemented as a mouse, a joystick, and the like.

[0042] The second human expert display 370 is employed for duplicating the human operator display 340 in the sense that it displays the same marked-up live video of the ultrasound scan probe 101 scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and the navigation pointer mark-ups. The second human expert display 370 can be implemented as a touch screen such that it effectively doubles as the human expert interface 360.

[0043] Figure 4 and Figure 5 show the fiducial marking arrangement 320 is preferably fashioned as an elongated split sleeve 321 made from relatively rigid material. The split sleeve 321 in its flattened form has a length L in the range of about 5 cm to 7 cm and a width W in the range of about 3 cm to 4 cm for forming an about 1 cm diameter closed split sleeve. The split sleeve 321 has a longitudinal split sleeve centerline 322 for being highly co-direction with the cable strain relief centerline 108 on placing the split sleeve 321 on the cable relief strain 107. On application of a flattening force, the split sleeve 321 assumes a generally rectangular flattened shape having a major edge pair 323A and 323B and a minor edge pair 324A and 324B transverse thereto.

[0044] The fiducial marking arrangement 320 preferably includes equally spaced apart fiducial marker pairs 326 for denoting fixed orientations with respect to the ultrasound scan probe 101. Each fiducial marker pair 326 has a unique visual distinctive characteristic, for example, a shape, a color, and the like. For assisting live tracking of the fiducial marking arrangement 320, the fiducial marking arrangement 320 preferable includes at least four fiducial marker pairs 326A- 326D. Each fiducial marker pair 326 preferably includes a fiducial marker adjacent the minor edge 324 A and a spaced apart fiducial marker adjacent the minor edge 324B. Figure 6 to Figure 9 show exemplary screen shots of a marked-up live video of an ultrasound scan probe scanning a patient’s region of interest with an ultrasound scan probe centerline mark-up and navigation pointer mark-ups as follows: Figure 6 shows an ultrasound scan probe centerline mark-up 400 and a circular ultrasound scan probe navigation mark-up 401 surrounding the ultrasound scan probe centerline mark-up 400. Figure 7 shows a tilt movement mark-up 402 relative to the ultrasound scan probe centerline mark-up 400 for instructing a human operator to tilt an ultrasound scan probe 101 relative to a patient’s region of interest. Figure 8 shows a down movement mark-up 403 relative to the ultrasound scan probe centerline mark-up 400 for instructing a human operator to depress an ultrasound scan probe 101 towards a patient’s region of interest. Figure 9 shows a clockwise rotation mark-up 404 relative to the ultrasound scan probe centerline mark-up 400 for instructing a human operator to rotate an ultrasound scan probe 101 relative to a patient’s region of interest.

[0045] Section 3: Tele-medicine set-up of mark-up apparatus with recording / storing functionality

[0046] Figure 10 shows a tele-medicine set-up 200B similar in construction and operation as the tele-medicine set-up 200A and therefore similar parts are likewise numbered. The mark-up apparatus 300 additionally includes a structured light source 380 for projecting structured light on a patient’s body including the patient’s region of interest for enabling the mark-up module 310 to determine the ultrasound scan probe 101 ’s 3D position relative to the patient’s body including deformation of the patient’s body due to downward pressure being applied to the ultrasound scan probe 101 for scanning purposes. The mark-up module 310 processes the stereo camera 330’s live stereo video of the ultrasound scan probe 101 scanning the patient’s region of interest for live identification of body landmarks, for example, a patient’s bellybutton, a patient’s birthmarks, and the like. During the course of an ultrasound scan, a human operator’s hand holding the ultrasound scan probe 101 typically obscures some but not all the ultrasound scan probe 101 from the stereo camera 330’s field of view. The mark-up module 310 identifies peripheral points of the ultrasound scan probe 101 for determining its 3D position in the 3D coordinate system. The mark-up apparatus 300 also includes a recording / storing module 390 for recording and storing ultrasound scans with clinical information for online and / or offline processing and analysis. The mark-up module 310 can include machine learning functionality for automatically overlaying navigation pointer mark-ups to the live video of the ultrasound scan probe scanning the patient’s region of interest. A human expert can approve the navigation pointer mark-ups, amend the navigation pointer markups or decline the navigation pointer mark-ups.

[0047] Section 4: Al set-up of mark-

[0048] Figure 11 shows an Al set 200C similar in construction and operation as the tele-medicine set-up 200B and therefore similar parts are likewise numbered and differing therefrom insofar as the Al set-up 200C includes a mark-up module 310 with functionality for substituting for a human expert. Accordingly, the Al set-up 200C does not include a first human expert display 350, a human expert interface 360 or a second human expert display 370.

[0049] Section 5 : Mark-up method of the present invention

[0050] Figure 12 is a flow diagram 500 of a mark-up method for assisting ultrasound scanning. The mark-up method includes the steps of

[0051] Step 501 : Providing a fiducial marking arrangement being manually securely placed on a cable strain relief for visually distinctively marking the cable strain relief

[0052] Step 502: Providing a stereo camera for acquiring a live stereo video of a ultrasound scan probe scanning a patient’s region of interest Step 503: Processing the live stereo video of the ultrasound scan probe scanning the patient’s region of interest for live tracking the fiducial marking arrangement for determining 3D orientation of the ultrasound scan probe centerline with respect to a 3D coordinate system for overlaying an ultrasound scan probe centerline mark-up to a live video of the ultrasound scan probe scanning the patient’s region of interest to create a marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up

[0053] Step 504: Overlaying at least one navigation pointer mark-up with respect to the ultrasound scan probe centerline mark-up to the marked-up live video for instructing a human operator how to manipulate the ultrasound scan probe relative to the patient’s region of interest

[0054] Step 505: Providing a human operator display for displaying the marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and the at least one navigation pointer mark-up.

[0055] While particular embodiments of the present invention are illustrated and described, it would be obvious to those skilled in the art that various other changes and modifications can be made without departing from the scope of the invention.

Claims

Claims:

1. Mark-up apparatus for use with an OEM ultrasound scanner having: i) an ultrasound scan probe having a longitudinal ultrasound scan probe centerline for placing on a patient’s region of interest for acquiring a live ultrasound scan, ii) an ultrasound scan controller including an ultrasound image display for displaying a live ultrasound image of the live ultrasound scan, and iii) an ultrasound scan cable for connecting the ultrasound scan probe and the ultrasound scan controller, the ultrasound scan cable having an elongated cable strain relief at its connection with the ultrasound scan probe, the cable strain relief having a longitudinal cable strain relief centerline highly co-directional with the longitudinal ultrasound scan probe centerline, the mark-up apparatus comprising:(a) a fiducial marking arrangement for being manually securely placed on the cable strain relief for visually distinctively marking the cable strain relief;(b) a stereo camera for acquiring a live stereo video of the ultrasound scan probe scanning the patient’s region of interest;(c) a mark-up module for performing the steps of: i) processing the live stereo video of the ultrasound scan probe scanning the patient’s region of interest for live tracking the fiducial marking arrangement for determining 3D orientation of the ultrasound scan probe centerline with respect to a 3D coordinate system for overlaying an ultrasound scan probe centerline mark-up to the live stereo video of the ultrasound scan probe scanning the patient’s region of interest to create a marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up, andii) overlaying at least one navigation pointer mark-up with respect to the ultrasound scan probe centerline mark-up to the marked-up live video for instructing a human operator how to manipulate the ultrasound scan probe relative to the patient’s region of interest; and(d) a human operator display for displaying the marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and the at least one navigation pointer mark-up.

2. The mark-up apparatus according to claim 1 wherein the fiducial marking arrangement includes at least four fiducial marker pairs spaced around the ultrasound scan probe’s cable strain relief wherein each fiducial marker pair includes a proximal fiducial marker adjacent the ultrasound scan probe and a distal fiducial marker spaced apart from the ultrasound scan probe.

3. The mark-up apparatus according to claim 2 wherein each fiducial marker pair of the at least four fiducial marker pairs has a unique shape.

4. The mark-up apparatus according to any one of claims 1 to 3 and further comprising a first human expert display in communication with the OEM ultrasound scanner for displaying the same live ultrasound scan as the OEM ultrasound scanner’s ultrasound image display, and wherein the mark-up module is configured to be operated by a human expert and includes: a human expert interface for enabling the human expert to overlay the at least one navigation pointer mark-up to the ultrasound scan probe centerline markup, and a second human expert display for displaying the same marked-up live video as the human operator display.

5. The mark-up apparatus according to any one of claims 1 to 4 and further comprising a structured light source for projecting structured light on the patient’s body including the patient’s region of interest for enabling the mark-up module to determine the ultrasound scan probe’s 3D position relative to the patient’s body.

6. The mark-up apparatus according to claim 5 wherein the mark-up module automatically overlays the at least one navigation pointer mark-up to the marked- up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up.

7. Mark-up method for use with an OEM ultrasound scanner having: i) an ultrasound scan probe with a longitudinal ultrasound scan probe centerline for placing on a patient’s region of interest for acquiring a live ultrasound scan, ii) an ultrasound scan controller including an ultrasound image display for displaying a live ultrasound image of the live ultrasound scan, iii) an ultrasound scan cable for connecting the ultrasound scan probe and the ultrasound scan controller, and the ultrasound scan cable having an elongated cable strain relief at its connection at the ultrasound scan probe, the cable strain relief having a longitudinal cable strain relief centerline highly co-directional with the longitudinal ultrasound scan probe centerline, the mark-up method including the steps of:(a) providing a fiducial marking arrangement being manually securely placed on the cable strain relief for visually distinctively marking the cable strain relief;(b) providing a stereo camera for acquiring a live stereo video of the ultrasound scan probe scanning the patient’s region of interest;(c) processing the live stereo video of the ultrasound scan probe scanning the patient’s region of interest for live tracking the fiducial marking arrangement for determining 3D orientation of the ultrasound scan probe centerline with respect to a 3D coordinate system for overlaying an ultrasound scan probe centerline markup to a live video of the ultrasound scan probe scanning the patient’s region of interest to create a marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up;(d) overlaying at least one navigation pointer mark-up with respect to the ultrasound scan probe centerline mark-up to the marked-up live video for instructing a human operator how to manipulate the ultrasound scan probe relative to the patient’s region of interest; and(e) providing a human operator display for displaying the marked-up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up and the at least one navigation pointer mark-up.

8. The mark-up method according to claim 7 wherein the fiducial marking arrangement includes at least four fiducial marker pairs spaced around the ultrasound scan probe’s cable strain relief wherein each fiducial marker pair includes a proximal fiducial marker adjacent the ultrasound scan probe and a distal fiducial marker spaced apart from the ultrasound scan probe.

9. The mark-up method according to claim 8 wherein each fiducial marker pair of the at least four fiducial marker pairs has a unique shape.

10. The mark-up method according to any one of claims 7 to 9 and further comprising the step of providing a first human expert display in communication with the OEM ultrasound scanner for displaying the same live ultrasound scan as the OEM ultrasound scanner’s ultrasound image display, andwherein the mark-up module is configured to be operated by a human expert and includes: a human expert interface for enabling a human expert to overlay the at least one navigation pointer mark-up relative to the ultrasound scan probe centerline mark-up, and a second human expert display for displaying the same marked-up live video as the human operator display.

11. The mark-up method according to any one of claims 7 to 10 and further comprising the step of projecting structured light on the patient’s body including the patient’s region of interest for enabling determining the ultrasound scan probe’s 3D position relative to the patient’s body.

12. The mark-up method according to claim 11 and further comprising the step of automatically overlaying at least one navigation pointer mark-up to the marked- up live video of the ultrasound scan probe scanning the patient’s region of interest including the ultrasound scan probe centerline mark-up.

Images