Imaging guidance systems and methods

Abstract

A first aspect provides a device for guiding a tool for transdermal treatment through an outer layer of a body of a subject. The device comprises an input for receiving a sensor signal from a transdermal sensor module, the sensor signal providing subdermal data on an organ below the outer layer of the subject, an image sensor arranged to capture image data from at least one of the outer layer and the needle, an electronic processing unit arranged to determine guidance data for projection of the guidance data on the outer layer, based on at least one of the subdermal data on the organ and the image data and a display module for displaying the guidance data. Such device enables a person other than a medical practitioner, for example a clinician, to perform a medical intervention on a person.

Description

IMAGING GUIDANCE SYSTEMS AND METHODSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 575,205, filed on 5 April 2024, the entire contents and substance of which is incorporated herein by reference in its entirety as if fully set forth below.TECHNICAL FIELD

[0001] The various aspects and examples thereof relate to a device for providing guidance for inserting on object, for example a needle, a catheter, a scalpel and the like, in a body of a living or deceased being, like a dead or living rat.BACKGROUND

[0002] With workload on medical practitioners increasing, work is being offloaded to clinicians and other staff supporting medical practitioners. Some staff may not be very skilled in certain operations, due to lack of theoretical knowledge, practical experience, or both.

[0003] For insertion of hollow needles for retrieving matter from or insertion of matter into the body of a living being, like a human, both theoretical knowledge and practical experience are required, using tools available today.SUMMARY

[0004] It is preferred to provide a tool with improved guidance on application of a tool to a body of a living being, for example on insertion of a needle or a scalpel.

[0005] A first aspect provides a device for guiding a tool for transdermal treatment through an outer layer of a body of a subject. The device comprises an input for receiving a sensor signal from a transdermal sensor module, the sensor signal providing subdermal data on an organ below the outer layer of the subject, an image sensor arranged to capture image data from at least one of the outer layer and the needle, an electronic processing unit arranged to determine guidance data for projection of the guidance data on the outer layer, based on at least one of the subdermal data on the organ and the image data and a display module for displaying the guidance data, for example, on at least one of the outer layer and the needle. Such device enables a person other than a medical practitioner, for example a clinician, to perform a medical intervention on a person, without the advance and elaborate training a medical practitionergenerally requires. And this device allows a medical practitioner to perform such work with less effort and higher accuracy than without guidance of this device. Transdermal is to be understood broadly; this also encompasses penetration of a sclera or cornea of an eye.

[0006] An implementation of the device further comprising an illumination module arranged to illuminate at least one of the needle, and / or the outer layer and wherein the image data comprises data on a shadow cast by the tool on the outer layer as a result of blocking light emitted by the illumination module to the outer layer. With a known position of the illumination module relative to a point of the needle, for example the tip, the shadow on the outer layer - the skin - provides data on a location and / or position of the needle. A known position of the tip may be the tip being placed on the skin. Alternatively or additionally, the image data on the shadow may also provide data on the location of the tip, as at the point where the tip touches the skin, the needle casts no shadow on the skin - or the shadow is at least not visible.

[0007] In another implementation, the processing unit is further arranged to determine, based on the image data, to determine an angle or other position data of the tool relative to a plane of the outer layer. With a known position of at least a part of the tool - like the tip of the needle - relative to the skin and a position of the device relative to the skin, a shadow may provide information on the angle of the tool relative to the skin; the position and optionally at least one of size and shape of the shadow may change as a function of an angle - at least one of azimuth and altitude - relative to the skin that changes.

[0008] In again another implementation, the illumination module comprises a first illumination source and a second illumination source spaced apart from the first illumination source. In this implementation, the processing unit is arranged to control the illumination source to illuminate the outer layer and the tool, detect, based on the image data, a first shadow of the tool on the outer layer provided by the first illumination source and a second shadow of the tool on the outer layer provided by the second illumination source; and determine the angle of the tool relative to the outer layer based on a position of the first shadow, a position of the second shadow and a position of the tool in the image data. With the illumination sources spaced apart at a distance from the skin, preferably more or less (+ / - 10%) the same distance, for example with a tip of a needle or scalpel set at the skin, an angle between a first shadow cast by virtue of the first light source and a second shadow cast by virtue of the second light source, changes as the altitude angle of the needle or scalpel changes relative to the skin. The light sources may be provided at substantially equal distances from the display module.

[0009] In a further implementation, the processing unit is arranged to determine organ location data on a position of the organ relative to the device, based on the subdermal data and determine, based on the location of the tool relative to the outer layer surface and the organ location data, altitude aim angle data providing an indication of an angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer.

[0010] The organ location data may be determined using, for example, ultrasound imaging data, though other technologies, like electromagnetic radiation (radar frequency, x-ray frequency, other, or a combination thereof) may be used as well. An ultrasound probe may to that purpose be integrate with or coupled to the device. Additionally or alternatively, other imaging technologies may be used. The position of the tool or a part thereof, like a tip of a scalpel or needle, may be determined as indicated above. Additionally or alternatively, a point where to position the tip may be prescribed, for example by projecting a marker on the skin. With the marker projected or tip detected at a particular position, location of the imaging probe known and depth of the organ known below the imaging probe, an angle for the tool to be positioned may be determined, for the tool to be moved for example in a substantially linear way into the skin and further into the body. The angle may subsequently be determined using common trigonometry.

[0011] In yet another implementation, the display module is arranged to display altitude guidance data on the outer layer, such as by a projection, the projected altitude guidance data providing an indication of an angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer. The guidance data may comprise one or more of a marker where a tip of a tool is to be placed, under what altitude angle and at what azimuth angle. The actual angles and positions may be determined as indicated above or in other ways.

[0012] In again a further implementation, the display module is arranged to project a first set of two lines equally spaced apart from the tool location relative to the outer layer as part of the projected altitude guidance data. By varying location of the lines as well as at least one of angle and distance between the lines, a significant amount of options is available for providing guiding data.

[0013] In again another implementation, the display module is further arranged to project a second set of two lines equally spaced apart from the tool location relative to the outer layer as part of the projected guidance data. In this device, the first set provides an indication of the angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer; the second set provides an indication of the angle of the tool relativeto the surface of the outer layer as detected and the first set of lines and the second set of lines are projected such that if the detected angle of the tool relative to the surface of the outer layer is substantially equal to the an angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer, at least one line of the first set coincides with at least one line of the first set. This example provides intuitive and direct user feedback in providing guidance.

[0014] In yet another implementation, the color of lines of at least one of the first set and the second set changes if the detected angle of the tool relative to the surface of the outer layer is substantially equal to the angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer. This implementation may of course be combined with any of the above and apart thereof, it provides direct and intuitive user guidance and feedback.

[0015] In a further implementation, the two lines of at least one of the first set and the second set cross at a point where the point of the tool is detected on the outer layer surface. As such, lines may coincide with the detected shadows provided in accordance with one of the implementations above. This provides clear, unambiguous, and intuitive user feedback and guidance.

[0016] In another implementation, the processing unit is arranged to determine, based on the image data, an angle of the tool in a plane parallel to the outer layer and relative to the device. This angle may also be considered as the azimuth angle (from Arabic J; as-sumut, "the directions"). This angle may also be determined using detection of shadow lines, as outlined above, or in different way using other imaging or other detection techniques. This data may be combined with data on detection of the organ to be reached by means of the tools, enabling guidance.

[0017] In again another implementation, the processing unit is arranged to receive organ location data on a position of the organ relative to the device, relative to the subdermal data and determine, based on the location of the tool relative to the outer layer surface and the organ location data, an azimuth aim angle data providing an indication of an angle of the tool in a plane parallel to the outer layer surface for the tool to reach the organ, when the tool is inserted through the outer layer. As indicated, this may provide input for proper projection of guiding data, in particular for guiding the azimuth of the tool relative to at least one of device, body part and skin.

[0018] In a further implementation, the display module is arranged to project azimuth guidance data on the outer layer, the projected azimuth guidance data providing an indicationof an angle of the tool in a plane parallel to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer. Such guiding data may provide more accurate and convenient guidance for proper insertion of the tool.

[0019] In again a further implementation, the display module is arranged to project a third set of two lines equally spaced apart from the tool location relative to the outer layer as part of the projected azimuth guidance data. As indicated, such set of two lines may provide clear user guidance how to insert the tool. In another implementation, actual altitude and azimuth are indicated by the same set of lines. A correct azimuth for insertion may be achieved if the lines indicating actual position are both at the same distance from corresponding target lines. Correct altitude may be achieved if the lines indicating actual position are at the same distance from one another as a distance between target lines. Lines, for actual positions and target positions are calculated and projected accordingly. As such, the second and third set may be the same.

[0020] In another implementation, the angle of the tool in a plane parallel to the outer layer for the tool to reach the organ is provided between the two lines of the third set. This provides clear and intuitive user feedback.

[0021] In a further implementation, the display module is further arranged to project a fourth set of at least one line, the line of the fourth set providing an indication of the detected angle of the tool in a plane parallel to the outer layer. Such may be a target line, alternatively or in addition to a set of two lines.

[0022] In yet another implementation, the color of lines of at least one of the third set and the fourth set changes if the detected angle of the tool in the plane parallel to the outer layer surface is substantially equal to an angle of the tool in the plane parallel to the outer layer surface for the tool to reach the organ, when the tool is inserted through the outer layer. Such projection of data provides clear and intuitive user guidance and user feedback.

[0023] In again a further implementation, the lines of each of the third set and the fourth set cross at a point where the point of the tool is detected on the outer layer surface. This provides a confirmation of where a position of the tooltip is detected - or is to be placed.

[0024] In again another implementation, the processing unit is arranged to determine, based on the image data and the subdermal data on the organ below the outer layer of the subject, how far the tool is to be inserted for the tool to reach the organ. If a particular position within the organ to be reached is important, this provides an advantageous additional to the general aspect. This is for example relevant if a needle is to be inserted in a vessel (vein, artery, lymph vessel) or a particular place in an organ.

[0025] In yet a further implementation, the display module is arranged to project data on at least one of the outer layer and the tool, data providing an indication how far the tool is to be inserted for the tool to reach the organ. This implementation provides instructions to the user, making work more convenient and the odds of error lower.

[0026] In yet another implementation, the processing unit is further arranged to determine, based on the image data, a proximal end of the tool; and control the display module to indicate, at a location of the tool, an indicator from the proximal end of the tool to a point on the tool that is provided directly above the outer layer when the tool is inserted in the outer layer and has reached the organ. This implementation provides direct and intuitive user feedback for accurate insertion of the tool to the correct and desired depth, in or on the organ.

[0027] In a further implementation, the electronic processing unit arranged to determine, based on the image data, at least one outer layer feature, determine, based on the determined outer layer feature within an image frame in which the image data is provided, a shift of the outer layer feature relative to the outer layer feature; and control the display module to adjust projection of guidance data on at least one of the outer layer in accordance with the detected shift. A living body moves, in particular if the subject is not sedated or under narcosis. As a result, a position of the device relative to the skin may change, which, in turn, results in changes of the projection on the skin. This implementation provides an option to correct projection parameters for such movements and may allow for projection of guidance data at the same location of the skin, irrespective of movements of the device relative to the skin.

[0028] In again a further implementation, the processing module is arranged to determine, based on the projection data and the image data, locations of the projected projection data on the outer layer and the outer layer feature relative to one another and control the display module to maintain the locations of the projected guidance data relative to the outer layer feature after the detected shift. This provides accurate and reliable guidance, irrespective of any movements.

[0029] In another implementation, the processing unit is arranged to determine, based on the subdermal data on the organ, a tool point location where to insert the tool under a predetermined angle to have the tool reach the organ when inserted under the predetermined angle; and control the display module to project, on the surface of the outer layer, the tool point location. This allows an assumption to be made where the tooltip is placed.

[0030] Alternatively or additionally, it provides a more or less proper point for the tooltip to be placed. In general, in particular in the case of needles, a tool is preferably placed under an angle of substantially 30° (+ / - 10%) relative to the skin place. If available, subdermal organlocation data may be used, if available and data on subdermal depth in particular, using trigonometry, where the tooltip is to be placed to reach the organ, with that tool placed under an angle of 30°.

[0031] In yet a further implementation, the electronic processing unit is arranged to determine a location of the organ, based on the subdermal data on the organ. Such data may be used in any of the implementations above.

[0032] Again another implementation further comprises a user input module for receiving input from a user to indicate, based on the image data, where the organ is located in the image data, wherein the subdermal data on a location of the organ is based on the user input. The user input can include haptics, a tap, a touch, a gesture, a voice, eye tracking, or other user inputs known to those of ordinary skill in the art. This allows to remove any ambiguity on what organ to reach, for example in a case where two vessels are located close to one another.

[0033] Again a further implementation further comprises a touch screen connector arranged to connected the device to a touch screen, wherein the electronic processing unit is further arranged to provide the image data to the touch screen for display; and receive user input data from the touch screen on where in the image data the organ is based. The user input data can include haptics, a tap, a touch, a gesture, a voice, eye tracking, or other user inputs known to those of ordinary skill in the art. This implementation makes it more convenient to make a selection of a specific organ to be reached and for which to provide guidance data.

[0034] In yet another implementation, the electronic processing unit is further arranged to, upon receiving the user input, provide a section of the image data to the touch screen, allowing the touch screen to provide a magnified view of the section of the image data corresponding to the user input. This enables a user to confirm and / or verify a selection of an organ, and / or to make selection more convenient.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The various aspects and details thereof will now be discussed in further detail in conjunction with drawings. In the drawings:

[0036] Figure 1 : shows a device placed on an arm;

[0037] Figure 2: shows a functional architecture of the device;

[0038] Figure 3: shows a sideview of the device placed on an arm;

[0039] Figure 4A: shows a device placed on an arm with a needle under a first angle;

[0040] Figure 4B: shows a device placed on an arm with a needle under a second angle;

[0041] Figure 5: shows alignment data projected on an arm;

[0042] Figure 6: shows alignment data projected on a needle;

[0043] Figure 7: shows skin features of an arm; and

[0044] Figure 8: shows a grid projected on an arm.DETAILED DESCRIPTION

[0045] Figure 1 shows a device 100 for guiding a needle 184 mounted on a syringe 180 for insertion of a needle tip 186 as a distal end of the needle 184 in an arm 190 or another part of a body, for example a human body. The needle tip 186 may be inserted in the arm 190, for the needle tip to be in contact with, for example inserted in, an organ of the person to which the arm belongs to extract matter from the organ or insert matter in the organ. Such organ may be, for example a vessel like a blood vein, a lymph vessel, or other. Alternatively, the needle 184 may be used to take a biopsy from an organ within the body, may be a tool to insert electronics or a battery related to a pacemaker or may be a scalpel.

[0046] The device 100 comprises an electronics housing 102 for housing electronics of the device 100, like an electronic processing unit and also comprises an electronic display screen 104, that may be implemented as a touch screen as a user input unit, for receiving user input. The device 100 further comprises a first infrared light source 142 and a second infrared light source 144, providing illumination units as part of an illumination module. The first infrared light source 142 and the second infrared light source 144 are spaced apart from one another. The infrared light sources are arranged to emit infrared light - also known as infrared electromagnetic radiation - with a wavelength of more than 800 nm, preferably more than 850 nm, towards the skin of the arm 190. In another example, an illumination module is provided that emits visible light.

[0047] The device 100 is also provided with a laser projector 126 as a display module. The laser projector is arranged to project visual data, visible to the human eye, on the skin of the arm 190. In one example, the laser projector 126 is arranged to project data in multiple colors, including at least two of red, green, blue and yellow. The laser projector 126 is in one example arranged to project image data on the skin, in an array of for example five by five centimetres. In another implementation, a projection source other than a laser may be used.

[0048] As would be understood, the device 100 can have other examples of a display module. For example, the device 100 can comprise a display screen as the display module. Alternatively, or in addition, the device 100 can comprise any output capable of displaying visual data, visible to the human eye, in a tangible medium. In the example provided above, the laser projector 126 outputs the visual data in a tangible medium (e.g., the skin of the arm190). In the example of a display screen, the display screen would output the visual data in a tangible medium (e.g., the screen).

[0049] Furthermore, the device 100 comprises a camera 132 with an electronic image capturing unit as an image sensor, that is optionally provided with a filter 134. The filter 134 is arranged to filter out light with a particular wavelength or wavelength range from any electromagnetic radiation that may enter the camera 132. The filter 134 is preferably arranged to let light with a wavelength as emitted by the infrared light sources is passed through. Light emitted by the laser projector 126 is preferably blocked by the filter 134. The camera 132 is preferably provided between the first infrared light source 142 and the second infrared light source 144.

[0050] The device 100 is connected to a handheld ultrasonic probe 112 that is connected to an ultrasonic data processing device 114. Other types of ultrasonic probes may be used as well. The ultrasonic data processing device 114 is arranged to control the ultrasonic probe 112 to emit an ultrasonic -acoustic - signal to the arm 190 and to process raw data, based on acoustic reflections received by the handheld ultrasonic probe 112, to provide image data. The image data provided is based on the acoustic reflections received and may provide an indication and a visual indication in particular, of any organs within the arm 190, like vessels, for example, blood.

[0051] The electronic display screen 104 is arranged to display the image data provided by the data processing device 114. The electronic display screen 104 and the handheld ultrasonic probe 112 may be part of the device 100, but may just as well be provided separately.

[0052] As would be appreciated, an ultrasound device typically incorporates a gel substance which is placed onto the patient’s skin order to ensure there is not air gap between the transducer and the patient’s skin, the gel also acts as a lubricant to make it easier to slide the transducer around on the patients skin in order to obtain the best view into the patient’s body.

[0053] The present disclosure can provide a transducer device which has a flat, low profile form factor, much like a ‘hockey puck’. The transducer can have an ultrasound array in its underside which is placed on the patient’s organ. A display can be embedded in the top of the device.

[0054] As such, the present disclosure can provide a ‘gel-pad’ uniquely shaped to conform to the profile of the transducer device. The gel-pad can be contained in a single-use package and can be designed to easily attach to the underside of the transducer, for instance, using a built- in adhesive layer that is part of the upper surface of the gel-pad. The gel-pad can be a standard thickness, providing a known ‘stand-off between the transducer and the patient’s organ. Thisstandoff can provide for a more optimized image if the gel-pad is used. The gel-pad may also be shaped with a concave lower surface, which can help it conform to the curved surface of the patient’s organ. The gel-pad may also have a sticky surface on its underside, potentially lightly adhering the transducer to the patient’s organ or vice versa.

[0055] Figure 2 shows a schematic representation of the device 100. The device 100 comprises an electronic processing unit 220. The electronic processing unit 220 is arranged to control the device 100 and the various components thereof and to control communication between the device 100 and other devices.

[0056] The electronic processing unit 220 is connected to an electronic memory 240. The electronic memory 240 is arranged to store data to be processed, processed data or a combination thereof. The electronic memory 240 is also arranged to have executable code stored thereon for programming the electronic processing unit 220 to perform various operations as disclosed below. The electronic memory may also store data processed by or to be processed by the electronic processing unit 220. The electronic memory 240 may be a volatile memory, a non-volatile - non-transitory - memory, or a combination thereof.

[0057] The electronic processing unit 220 is further connected to an illumination module 140 that comprises, in the example of Figure 1, the first infrared light source 142 and the second infrared light source 144. The electronic processing unit 220 is also connected to an imaging unit 130 comprising the camera 132, to the projection unit 126, the electronic display screen 104 and to the handheld ultrasonic probe 112. The connection between the electronic processing unit 220 and the handheld ultrasonic probe 112 may be provided via the ultrasonic data processing device 114.

[0058] The first infrared light source 142 and the second infrared light source 144 can project light in the 700 - 940 nm range onto the organ. Because hemoglobin in veins absorbs more light than surrounding tissues, the vessels (e.g., arteries and veins) will show as darker areas under the skin. In the case where the target organ is a vein, a darker area under the skin will align with the targeted organ in the display. The laser line can also likely align with one of the darker areas under the skin. This alignment will reinforce to the user that the system is providing guidance that is in concordance with the vessel structures that are under the skin, but outside the area of skin that is under the ultrasound transducer element.

[0059] Figure 3 shows a sideview an example of how the needle 184 mounted on the syringe 180 is ideally placed relative to the arm 190 and in particular relative to a vein 192 as an organ within a body. In this drawing, an arm is depicted; in other examples, the needle or other object may be inserted in another limb or another part of the body to reach a vein as well - or anotherorgan. The handheld ultrasonic probe 112 is placed above the vein 192, that lies at a distance D below the surface of the skin of the arm 190. The needle 184 is ideally placed at an angle A of 30°, relative to the skin of the arm 190 - though other angles may be envisaged as well, for example between 15° and 45°, or for example between 20° and 30° or for example between 25° and 35°.

[0060] The needle tip 186 is placed at a distance L from the handheld ultrasonic probe 112. By specifying, by the device 100, as disclosed below in conjunction with Figure 5, where a user - clinician, medical practitioner or other person - is to place the needle tip 186, the distance L is known. The distance D may be determined based on ultrasonic data received from the handheld ultrasonic probe 112. With L and D known, insertion length I may be calculated, with I being the length by which the needle 184 is to be inserted to reach the vein.

[0061] Figure 4 A shows a first isometric view of the needle 184 being placed on the arm 190 under a first angle. The dashed lines from the infrared light source to the needle 184 and to the arm 190 indicate light that is blocked by the needle 184, which results in shadows on the arm 190. Light from the first infrared light source 142 that is blocked results in a first shadow 402 and light from the second infrared light source 144 that is blocked by the needle results in a second shadow 404.

[0062] Figure 4 B shows a second isometric view of the needle 184 being placed on the arm 190 under a second angle. The second angle is smaller than the first angle. Light from the first infrared light source 142 that is blocked results in a third shadow 402' and light from the second infrared light source 144 that is blocked by the needle results in a fourth shadow 404'.

[0063] The smaller the angle of the needle 184 relative to the skin of the arm 190, the smaller the angle between the shadows, as can be seen from the differences between the constellation depicted by Figure 4 A and the constellation depicted by Figure 4 B. This means that from the angle between the shadows, for example between the first shadow 402 and the second shadow 404, the angle of the needle 184 relative to the skin of the arm - the altitude or altitude angle of the needle 184 - may be determined. The shadows on the arm 190 may be detected and identified by means of the electronic processing unit 220, by processing image data captured by means of the camera 132. An angle of the needle 184 in a plane parallel to the skin of the arm 190 - the azimuth of the needle 184 - which basically coincides with the x-y plane identified in Figure 4 A, may be determined based on image data captured by means of the camera 132 as well, as the needle 184, generally made comprising metal, reflects light as well as (near) infrared light or (near) infrared radiation in a different way than skin does.

[0064] With the possibility to determine the angle between the needle 184 and the skin of the arm 190 and the angle of the needle 184 in a plane substantially parallel to the skin of the arm 190, the position of the needle 184 relative to the arm 190 may be determined. Furthermore, with the distance L (Figure 3) basically known, also the location of the needle 184 relative to the arm and relative to the device 100 and relative to the handheld ultrasonic probe 112 may be known. Additionally, with the position of the vein 192 known relative to the device 100 and relative to the handheld ultrasonic probe 112, also the position of the needle 184 and the needle tip 186 relative to the vein 192 may be known, by processing image data and other data by means of the electronic processing unit 220.

[0065] Alternatively, or in addition, the camera 132 can capture a view of the needle 184 that is entering the organ. The camera 132 can be orthogonal to the organ such that if the organ was visible in the image it would appear as a horizontal plane, with x being the distance ‘near / far’ in the camera’s view, and y being the distance Teft / right’ in the camera’s view, and z referencing the location of the needle 184 in the vertical axis ‘above / below’ the skin. The distal end of the needle 184 can be pointing towards the camera 132 in the x dimension. A computer vision algorithm can be used to detect the needle 184 in the image as the needle 184 will appear as a solid vertical line in the image. The needle 184 is being held at an angle to the organ, and in order for the needle 184 to penetrate downwards, the needle tip will end up below the organ meeting a desired end point (for the purpose of entering a vessel, placing a drug, extracting a sample from the body, or implanting something at a desired end point in the body). Because the needle 184 is at an angle, the image of the needle 184 will have a ‘keystone’ appearance, meaning the lower width of the needle 184 (closer to the camera 132) will be smaller than the upper width of the needle 184 (further away from the camera 132). An algorithm can calculate the angle the needle 184 is at based on the keystone detected in the image. This information can be used in a needle guidance algorithm to then provide feedback to the user to adjust the trajectory of the needle 184 so it will be inserted at the correct angle needed to have the needle tip end at the desired ‘end point’ within the patient’s body.

[0066] By way of another example, since the camera 132 in the device 100 has a known depth of field that can be controlled by a combination of focal length and aperture size, the needle 184 trajectory can be determined by focus mechanisms.

[0067] An in-focus needle edge will always be high contrast. As a needle edge de-focuses, the edge contrast becomes measurably softer (in other words, the transition from background to needle colour may take 20 pixels rather than 2).

[0068] If the needle tracking optics are adjusted such that the needle 184 in the horizontal plane on the organ surface will de-focus at distance k from the edge of the device 100 in the camera field of view, then as the needle 184 is tilted in the z axis, the distance of parts of the needle shaft with respect to the camera lens reduces, and the apparent de-focus distance along the needle changes to k-m, where m is determined by needle insertion coordinates, z-axis tilt, and camera position and orientation, where z-axis tilt is the only variable once the needle tip has made contact with the organ at the insertion point.

[0069] An algorithm can calculate the angle the needle 184 is at based on the focus and an algorithm of this type. This information can be used in a needle guidance algorithm to then provide feedback to the user to adjust the trajectory of the needle 184 so it can be inserted at the correct angle needed to have the needle tip end at the desired ‘end point’ within the patient’s body.

[0070] It is also understood that a comprehensive approach could exploit both keystoning and focus mechanisms from the same optical configuration in order to more accurately assess needle position so the needle 184 can be inserted at the correct angle needed to have the needle tip end at the desired ‘end point’ within the patient’s body.

[0071] Figure 5 shows a top view of the skin of the arm 190, with the needle tip 186 placed thereon. By means of the laser projector 126, a circle 532 is projected on the skin as a needle tip marker, though other shapes may be used as well, like a cross, a square, a triangle, other, or a combination of two or more thereof. The needle tip marker is provided to provide an indication to a user where to place the needle tip 18 6 on the skin. As indicated above, the needle tip 186 is preferably placed such that the needle 184 reaches, when linearly inserted into the arm, the vein 192 just below the ultrasonic probe 112.

[0072] Prior to projecting the circle 532, the handheld ultrasonic probe 112 is to be placed on the arm 190. When placed on the arm, the handheld ultrasonic probe 112 is used to provide image data of the arm 190 below the handheld ultrasonic probe 112. This is depicted by Figure 1, in which an image 192' of the vein 192 is displayed on the electronic display screen 104.

[0073] The user may select the vein 192 by tapping on the image 192' of the vein 192 at the electronic display screen 104, as a touch screen.

[0074] The processing unit 220 determines the depth D of the vein 192 (Figure 1), based on the ultrasonic imaging data received from the handheld ultrasonic probe 112, optionally via the ultrasonic data processing device 114. And based on the depth D and the 'ideal' angle of insertion A (Figure 1) being 30°, an insertion point may be determined where the needle tip 18 6 is to be placed, for example, on the skin of the arm 190. At that location, the circle 532 isprojected by means of the laser projector 126 or other display means, like a display screen. Once the needle tip 186 is placed on the skin, this may be detected based on image data captured by means of the imaging unit 130. Upon detection of the needle tip 186 having been placed on the skin, the laser projector 126 projects a first altitude reference line 502 and a second altitude reference line 504. The altitude reference lines serve as guidance for providing an altitude of the needle 184 - which is the angle of the needle 184 relative to the skin of the arm 190.

[0075] The laser projector also provides, in this example, a first needle altitude line 512 and a second needle altitude line 514. The needle altitude lines provide an indication of the altitude of the needle 184, i.e. the angle of the needle relative to the skin of the arm 190. In one example, the needle altitude lines 512 coincide with the shadows of the needle, which shadow are discussed in conjunction with Figure 4 A and Figure 4 B. In one example, the altitude of the needle 184 is correct for insertion of the needle for reaching the vein 192 if the needle altitude lines meet with the altitude reference lines.

[0076] Figure 5 shows that the altitude reference lines and the needle altitude lines cross or meet at the circle 532. In one example, the needle altitude lines are provided equidistantly relative to a desired azimuth of the needle 184. In another example, the needle altitude lines are provided equidistantly relative to an actual or desired azimuth of the needle 184. In a further example, reference altitude lines are provided equidistantly relative to a desired azimuth of the needle 184. In this implementation, a position of the needle 184 as a tool to be applied is indicated in terms of angles. In another implementation, additionally or alternatively, other coordinate systems may be used, like distances or Cartesian coordinates.

[0077] In again another example, the altitude reference lines are provided equidistantly relative to an actual azimuth of the needle 184. A combination of different of these four example may be used, with, for example, reference altitude lines provided equidistantly relative to a desired azimuth of the needle 184 and needle altitude lines provided equidistantly relative to an actual azimuth of the needle 184.

[0078] Figure 5 also shows an azimuth reference line 522 projected by the laser projector 126 on the skin of the arm 190. The azimuth reference line 522 indicates in this example the ideal angle of the needle 184 in a (geometrical) plane parallel to the skin - the azimuth - for the needle tip 186 to reach the vein 192 with the needle inserted in the arm 190, inserted under that ideal angle. In another example, the azimuth reference line 522 may coincide with the actual - detected - position of the needle 184. Hence, the azimuth reference line 522 may be projected on the needle 184. Once an indication that the azimuth of the needle 184 is as desired, a colorof the azimuth reference line 522 may change upon reaching the ideal insertion azimuth. For example, the color may change from red to green, optionally passing through orange, to indicate a slow change an approach towards a target angle. It is noted that also other color schemes may be used.

[0079] Once the needle tip 188 is aligned with the circle 532 and with the desired altitude and azimuth, the laser projector 126 provides a needle position marker 602 and a needle destination marker 604 on the needle 184, on a display screen or another object, as depicted by Figure 6 A and Figure 6B. Figure 6 A shows the needle 184 not inserted, with the needle tip 186 located in the projected circle 532. In this example, the needle position marker 602 is provided between a needle hub 188 of the needle 184 and a location of the needle 184 that is not inserted when the needle 184 has been inserted at the preferred depth I (Figure 3). For example, the needle position marker 602 is provided at the needle 184 at a position close to the needle hub 188 - or another part of the needle or syringe, like the needle tip. Subsequently, the needle 184 is to be inserted in the arm 190. While the needle is being drawn into the arm 190, the position of the needle hub 188 as proximal end of the needle 184 is followed by processing image data captured by means of the imaging unit 130. The laser projector continues to project the needle position marker 602 on the needle 184 at a fixed distance relative to the needle hub 188. The part of the needle 184 inserted in the arm 190 is indicated with dashed lines, though other visual characteristics may be used as well.

[0080] The laser projector also projects the needle destination marker 604 on the needle 184 or close to the needle, at a fixed distance relative to the projection of the circle 532. The needle destination marker 604 and the needle position marker 602 are provided such that when they overlap, for example fully overlap, the needle tip 186 reaches the vein 192 or another object in the body to be reached, like a bone or an organ. Alternatively, the needle destination marker 604 and the needle position marker 602 are provided such that when they fully overlap, the needle tip 186 has punctured and is located in the vein 192. Different cases may be implemented depending on user preferences.

[0081] It is possible that during operation of the device 100, the arm 190 may move and the position of the device 100 may change relative to the arm. With the laser projector 126 being located on the device 100, this may result in the location of any data projected on the skin of the arm to change as well, relative to the arm. This may, in turn, require the user to start the procedure as described above again.

[0082] To address moving of the device 100 relative to the arm 190, image data comprising data of features of the skin of the arm 190 captured by means of the imaging unit 130 may beprocessed to determine and identify the features of the skin - skin features. Such features may be hair, freckles, ridges, bumps, other, or a combination of two or more thereof. Whereas such features may not be as simple to detect as finger prints, such skin features do have unique characters. As such, these features may be used for tracking of skin.

[0083] Figure 7 shows the arm 190, with the device 100 comprising the laser projector 126 being placed on top of the ultrasonic probe 112. Figure 7 shows, by means of a dashed line, an image capturing field 710 of which the imaging unit 130 captures image data. The capturing of the data results in an image 720, of which a close up image 730 is provided by Figure 7 as well. Within the image 720, skin features likes ridges and troughs are detected and identified, for example based on shape of the features. If the device 100 moves relative to the arm 190, the identified features move within the imaging field 710.

[0084] The shift of the identified features provides an indication of movement of the device 100 relative to the skin of the arm 190. Based on the shift of the features, the electronic processing unit 220 may control the laser projector 126 - or other electronic display means - to shift projection of one or more markers discussed above to compensate for the movement of the device 100. In this way, for example once the needle tip 186 has been placed on the skin of the arm 190. In this way, projection of the circle 532 is maintained at a fixed position on the skin of the arm 190. In an alternative, this action is performed when the needle tip is not on the skin, but already close to the skin.

[0085] An arm, but also other member of a body, in particular of smaller subject, like infants, have surfaces that are not perfectly flat. Most members are curved in one way or the other. This may result in distortion of image data obtained from the member, by the imaging unit 130. This applies to shadows of the needle, as well as to skin features. Furthermore, such curvature may also result in distortion of image data projected on the member of the body, like a limb. It may be advantageous to compensate for such distortions.

[0086] To determine distortions caused by curvatures of a body or body members, a grid 810 as shown by Figure 8 may be projected on, for example, the arm 190. With the grid 810 being provided as having a square or rectangular shape - or another pre-determined, preferably regular, shape, with cells having substantially the same size and shape, the actual projection may be distorted due to the curves of the limb. The grid 810 may be displayed using visible light or infrared light, also known as infrared radiation.

[0087] The grid as projected may be captured as image data by means of the imaging unit 130. If the imaging unit 130 is provided with the low pass filter 134 (Figure 1), it is preferredthe grid 810 is projected using infrared light with a wavelength or spectrum having a wavelength of more than 700 nm and preferably not more than 850 nm.

[0088] The projection of the grid 810 may be provided by the laser projector 126 or a dedicated grid projector, comprised by the device 100. In another example, visible light is used for proj ection of the grid 810 and either the low pass filter 134 is omitted or a dedicated imaging unit for capturing visible light is provided on the device 100.

[0089] The distortion of the grid 810 may be detected and determined by means of the electronic processing unit 220 and reference grid data that may, for example, be stored in the electronic memory 240. Subsequently, based on the detected and determined distortion, a transformation model may be determined to compensate for the distortion. The transformation model may subsequently be used to reconstruct shadows cast by the needle 184 and project any reference markers on the skin of the arm 190 or another limb such that the marker appears straight and accurate, from above. And the transformation model may also be used to transform guidance data to be project to compensate for the distortion.

Claims

CLAIMSWhat is claimed is:

1. A device for guiding a tool for transdermal treatment through an outer layer of a body of a subject, the device comprising: an input for receiving a sensor signal from a transdermal sensor module, the sensor signal providing subdermal data on an organ below the outer layer of the subject; an image sensor arranged to capture image data from at least one of the outer layer and the tool; an electronic processing unit arranged to determine guidance data for projection of the guidance data on the outer layer, based on at least one of the subdermal data on the organ and the image data; a display module for displaying guidance data.

2. The device according to claim 1, further comprising an illumination module arranged to illuminate at least one of the tool and the outer layer and wherein the image data comprises data on a shadow cast by the tool on the outer layer as a result of blocking light emitted by the illumination module to the outer layer.

3. The device according to any one of claim 1 or claim 2, wherein the processing unit is further arranged to, based on the image data, determine a location of the tool relative to the outer layer surface.

4. The device according to any one of claim 1 to claim 3, wherein the processing unit is further arranged to, based on the image data, determine a position of the tool relative to the outer layer surface.

5. The device according to any one of claim 1 to 4, wherein the processing unit is further arranged to detect, based on the image data, a location a distal end of the tool on the outer layer.

6. The device according to any one of claims 1 to 5, wherein the processing unit is further arranged to determine, based on the image data, an angle of the tool relative to a plane of the outer layer.

7. The device according to claim 2, the illumination module comprising a first illumination source and a second illumination source spaced apart from the first illumination source, wherein the processing unit is arranged to: control the illumination source to illuminate the outer layer and the tool; detect, based on the image data, a first shadow of the tool on the outer layer provided by the first illumination source and a second shadow of the tool on the outer layer provided by the second illumination source; and; determine an angle of the tool relative to the outer layer based on at least one of a position of the first shadow, a position of the second shadow and a position of the tool in the image data.

8. The device according to any one of the claims 1 to 7, wherein the processing unit is arranged to: determine organ location data on a position of the organ relative to the device, based on the subdermal data; determine, based on the location of the tool relative to the outer layer surface and the organ location data, altitude aim angle data providing an indication of an angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer.

9. The device according to any one of the claims 1 to 8, wherein the display module is arranged to project altitude guidance data on the outer layer, the projected altitude guidance data providing an indication of an angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer.

10. The device according to claim 9, wherein the display module is arranged to project a first set of two lines equally spaced apart from a location of the tool relative to the outer layer as part of the projected altitude guidance data.

11. The device according to claim 10, wherein the display module is further arranged to project a second set of two lines equally spaced apart from a location of the tool relative to the outer layer as part of the projected guidance data, wherein: the first set provides an indication of the angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer; the second set provides an indication of the angle of the tool relative to the outer layer as detected; the first set of lines and the second set of lines are project such that if the detected angle of the tool relative to the outer layer is substantially equal to the an angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer, at least one line of the first set coincides with at least one line of the first set.

12. The device according to any one of claim 10 or 11, wherein a color of lines of at least one of the first set and the second set changes if the detected angle of the tool relative to the outer layer is substantially equal to the angle of the tool relative to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer. Modification of saying Tine color’ to possibly include color coding angle and trajectory display screen parameters ‘as seen above’.

13. The device according to any one of claim 10 or 12, wherein the two lines of at least one of the first set and the second set cross at a point where the point of the tool is detected on the outer layer surface.

14. The device according to any one of the claims 1 to 13, wherein the processing unit is arranged to determine, based on the image data, an angle of the tool in a plane parallel to the outer layer and relative to the device.

15. The device according to any one of the claims 1 to 14, wherein the processing unit is arranged to: receive organ location data on a position of the organ relative to the device, relative to the subdermal data;determine, based on the location of the tool relative to the outer layer surface and the organ location data, an azimuth aim angle data providing an indication of an angle of the tool in a plane parallel to the outer layer surface for the tool to reach the organ, when the tool is inserted through the outer layer.

16. The device according to any one of the claims 1 to 15, wherein the display module is arranged to project azimuth guidance data on the outer layer, the projected azimuth guidance data providing an indication of an angle of the tool in a plane parallel to the outer layer for the tool to reach the organ, when the tool is inserted through the outer layer.

17. The device according to claim 16, wherein the display module is arranged to project a third set of two lines equally spaced apart from a location of the tool relative to the outer layer as part of the projected azimuth guidance data.

18. The device according to claim 17, wherein the angle of the tool in a plane parallel to the outer layer for the tool to reach the organ is provided between the two lines of the third set.

19. The device according to any of claim 16 to claim 18, wherein the display module is further arranged to project a fourth set of at least one line, the line of the fourth set providing an indication of the detected angle of the tool in a plane parallel to the outer layer.

20. The device according to any one of claims 16 to 19, wherein a color of lines of at least one of the third set and the fourth set changes if the detected angle of the tool in the plane parallel to the outer layer surface is substantially equal to an angle of the tool in the plane parallel to the outer layer surface for the tool to reach the organ, when the tool is inserted through the outer layer.

21. The device according to any one of claim 16 to 20, wherein the lines of each of the third set and the fourth set cross at a point where the point of the tool is detected on the outer layer surface.

22. The device according to any one of the claim 5, wherein the processing unit is arranged to determine, based on the image data and the subdermal data on the organ below the outer layer of the subject, how far the tool is to be inserted for the tool to reach the organ.

23. The device according to claim 22, wherein the display module is arranged to project data on at least one of the outer layer and the tool, data providing an indication how far the tool is to be inserted for the tool to reach the organ.

24. The device according to claim 23, wherein the processing unit is further arranged to: determine, based on the image data, a proximal end of the tool; and control the display module to indicate, at a location of the tool, an indicator from the proximal end of the tool to a point on the tool that is provided directly above the outer layer when the tool is inserted in the outer layer and has reached the organ.

25. The device according to any one of the claims 1 to 24, wherein the image data comprises image frames provide consecutive in time and the electronic processing unit arranged to: obtain a first image frame comprised by the image data; determine, based on the first image frame , a first location of at least one first outer layer feature; obtain a second image frame comprised by the image data; determine a second location of the at least one first outer layer feature; determine, based on the first location and the second location, a shift of the outer layer feature; and control the display module to adjust projection of guidance data in accordance with the detected shift.

26. The device according to claim 25, wherein the processing unit is arranged to: determine, based on the projection data and the image data, locations of the projected projection data on the outer layer and the outer layer feature relative to one another; and control the display module to maintain the locations of the projected guidance data relative to the outer layer feature after the detected shift.

27. The device according to any one of claims 1 to 26, wherein the processing unit is arranged to: determine, based on the subdermal data on the organ, a tool point location where to insert the tool under a predetermined angle to have the tool reach the organ when inserted under the predetermined angle; and control the display module to project, on the surface of the outer layer, the tool point location.

28. The device according to any one of the claim 1 to 27, wherein the electronic processing unit is arranged to determine a location of the organ, based on the subdermal data on the organ.

29. The device according to any one of the claim 1 to 27, further comprising a user input module for receiving input from a user to indicate, based on the image data, where the organ is located in the image data, wherein the subdermal data on a location of the organ is based on the user input.

30. The device according to claim 29, further comprising a touch screen connector arranged to connected the device to a touch screen, wherein the electronic processing unit is further arranged to: provide the image data to the touch screen for display; and receive user input data from the touch screen on where in the image data the organ is based.

31. The device according to claim 30, wherein the electronic processing unit is further arranged to, upon receiving the user input, provide a section of the image data to the touch screen, allowing the touch screen to provide a magnified view of the section of the image data corresponding to the user input.

32. The device according to any one of the claims 30 to 31, further comprising the touch screen.

33. The device according to any one of the claims 1 to 32, further comprising the transdermal sensor module.

34. The device according to any one of the claims 1 to 33, wherein the transdermal sensor module is an ultrasonic echography imaging device.

35. The device according to any one of claims 1 to 34, further comprising: a gel pad for use with an ultrasound device; and a flexible base layer positioned on a bottom surface of the ultrasound probe.

Images