System and method for acquiring thermal image data of body parts and thermal imager

Abstract

A computer-implemented method for acquiring thermal image data of a body part and a thermal imager are disclosed. The method includes: receiving image data of a body part, the image data including thermal image data, the thermal image data including thermal image data of the thermal image data of the body part and background thermal image data; applying a classifier to the image data to detect a contour of the body part, the classifier being configured to use image data other than the thermal image data for detection; generating a mask from the contour, applying the mask to the thermal image data to remove the background thermal image data and obtaining the thermal image data of the body part.

Description

Technical Field

[0001] The present invention relates to a system and method for acquiring thermal image data of a body part and a thermal imager, and is particularly suitable for identifying diabetic foot ulcers. Background Art

[0002] Patients with diabetes are prone to severe foot ulcers that may become infected and ultimately require amputation. Early detection is critical to improving patient outcomes, allowing doctors to intervene to protect the foot before skin breakdown occurs and to administer antibiotics promptly in the event of infection.

[0003] It is estimated that up to 50% of people with diabetes experience foot neuropathy, which can lead to extensive skin damage that goes undetected, leading to ulcers (diabetic foot ulcers (DFUs)), infection and amputation. This chain of events is largely avoidable if early preventive measures are taken, with an estimated 80% reduction in amputation rates. Mortality following DFUs and amputations is high, with up to 70% of people dying within 5 years of amputation and around 50% of people dying within 5 years of developing a diabetic foot ulcer. In addition to the adverse effects on patients, DFUs and complications are estimated to cost the National Health Service in England and Wales more than £1 billion per year.

[0004] Medical research has shown that skin temperature can rise by more than 2°C for up to 10 days before skin breakdown occurs. Often, skin temperature rises before any visible signs of infection or abnormality appear on the feet. This is not necessarily related to infection, but can also be caused by tissue breakdown caused by friction and the subsequent increased blood flow to the area.

[0005] Achieving the necessary accuracy in measurements and being able to monitor the temperature of the foot over time is a particular challenge.

[0006] Most physicians routinely perform visual inspection, tactile sensitivity testing, and palpation of the feet of patients with diabetes to detect local temperature changes that may indicate early lesions (pre-ulcers) in patients with diabetes. These manual techniques are notoriously inaccurate and can be supplemented with thermography (contacting the foot with a measuring device or platform to take temperature measurements) or thermal imaging diagnostic techniques where ulcers are present. However, readily available commercial thermography and thermal imaging devices and techniques also have various drawbacks.

[0007] First, no known device has a size, cost, and ease of use suitable for use outside of a large medical facility or laboratory.

[0008] Ideally, such equipment should be available for use by podiatrists, nursing homes, etc. without the need for a skilled operator.

[0009] Second, even with a skilled operator, existing devices are unable to provide temperatures with clinically relevant uncertainties (i.e., better than ±0.3°C (k=2)). For example, many DFU current thermography devices require that the patient’s foot is in physical contact with the device. In this case (contact-based measurement), the temperature and thermophysical properties of the device interfere with the temperature of the object (the foot), and therefore affect the accuracy of the measurement.

[0010] Even when other potential differences and errors have been minimized or eliminated, it has been determined that the accuracy of commercial off-the-shelf thermal imagers for absolute temperature measurements varies widely. In a study (Whittam et al., “Performance Testing of Thermal Imaging Systems to Assess Their Suitability for Quantitative Temperature Measurements”, presented at the 12th International Conference on Quantitative Infrared Thermography, Bordeaux, France, July 7-11, 2014),

[0011] https: / / www.ndt.net / search / docs.php3?showForm=off&id=17768 ), 5 of 6 thermal imaging devices were identified as having temperature measurement errors outside the manufacturer's own specified error range at 1 meter from the measurement target. Assuming that the manufacturer's specified error range is typically ±2°C, and that thermal imaging devices cannot achieve this error range repeatedly and uniformly, they cannot provide clinically relevant temperatures (uncertainty <±0.3°C), and it is highly questionable whether they can be relied upon to issue early warnings of sustained increases in skin temperature >2°C.

[0012] Reliable foot temperature comparisons over time require accurate spatial registration of the foot each time an image is acquired. This can be challenging because operators (and the abilities and methods they employ) can vary, and the position and size of the foot can vary between patients and between measurements, distorting the accuracy of foot temperature measurements.

[0013] Therefore, there seems to be considerable room for improvement in both equipment and methods. If the temperature increase associated with neuropathy can be identified promptly and accurately, ulcers could be greatly reduced by, for example, prophylactic unloading. Summary of the invention

[0014] According to one aspect of the present invention, there is provided a computer-implemented method for acquiring thermal image data of a body part, comprising:

[0015] receiving image data of a body part, the image data comprising thermal image data, the thermal image data comprising thermal image data of the body part and background thermal image data;

[0016] applying a classifier to the image data to detect a contour of the body part, the classifier being configured to perform the detection using image data other than the thermal image data;

[0017] generating a mask according to the contour; and

[0018] The mask is applied to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.

[0019] Embodiments of the present invention concern a method and system in which additional data is used to guide the extraction of thermal images of a body part. The additional data may be other image data of the same scene (e.g., visual images or stereoscopic images). The additional data may also (or alternatively) include training data that has been used to train a machine learning or artificial intelligence system, such as a neural network, a genetic algorithm, etc. For example, a training library of foot images (visual, thermal, other image types, or some combination) may be used to train a classifier to determine contours from displayed image data, which can then be used to generate a mask and extract thermal image data of the body part. In a preferred embodiment, the system is used in a handheld thermal imaging system to identify potential or predicted foot ulcers from captured thermal images.

[0020] In thermal imagers, it is often assumed that foreground imaged objects will be much hotter than the surroundings (background), so background subtraction can be applied. However, this assumption does not always hold. The background temperature is not guaranteed to be very different from the foreground body parts, and may vary depending on the time of day, season, or environment (for example, it is not uncommon in hospitals and nursing homes, and embodiments of the present invention can be used to have relatively high background temperatures to provide comfort for people with limited mobility or who are bedridden). In addition, DFUs are common around the big toes and soles of the feet. Where such areas are cooler (for example, due to circulation issues), there is a risk that the foot area will not be properly distinguished from the background. Embodiments of the present invention attempt to address this issue by utilizing additional image data to more clearly distinguish the edges of body parts.

[0021] It should be understood that the methods may be implemented in parallel or in combination, for example, the neural network generates what are considered contours from thermal image data, which is in contrast to the contours generated by the machine vision system from a stereo image of the scene. Alternatively, the neural network may operate on the stereo image and then apply the generated contours to the thermal image. Alternatively, the neural network may operate on the stereo image and then apply the generated contours to the thermal image.

[0022] The image data may include visual image data captured from the same scene as the thermal image data, the classifier being configured to detect a contour of the body part from the visual image data.

[0023] The image data may include visual stereo image data captured from the same scene as the thermal image data.

[0024] The classifier may include an artificial neural network or other artificial intelligence system trained from image data of the body part to detect the contours of the body part.

[0025] The classifier may be configured to determine a spatial orientation of the body part, the method further comprising the step of modifying a spatial orientation of the obtained thermal image data of the body part to match a default spatial orientation based on the determined spatial orientation.

[0026] Although the current focus is primarily on thermal imaging for detecting the onset of neuropathy in the feet and triggering advanced preventive measures, the methods described in this application are also applicable to other body parts of human and / or animal subjects, such as: eyes, limbs, and other body parts.

[0027] The inventors of the present application have conceived a non-contact thermal imaging device, system, and method that address many of the shortcomings of known thermal imaging diagnostic devices and systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Embodiments of the present invention are described below by way of example only with reference to the accompanying drawings, in which:

[0029] Figure 1 is a schematic diagram of a thermal imager according to an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a thermal image processing system according to an embodiment of the present invention;

[0031] Figure 3 is a more detailed description Figure 2 Schematic diagrams of various aspects of the system;

[0032] Figure 4 is a more detailed description Figure 2 Schematic diagrams of various aspects of the system;

[0033] Figure 5 is a more detailed description Figure 2 Schematic diagram of optional aspects of the system. DETAILED DESCRIPTION

[0034] Figure 1 is a schematic diagram of a thermal imager according to an embodiment of the present invention.

[0035] The thermal imager 10 is preferably a handheld form factor that includes a thermal camera 20 , a battery power source 30 , a processor 40 , a data repository 50 , and a touch screen display 60 .

[0036] The thermal imager also includes a thermal image processing system 100 executed by the processor 40 to obtain thermal image data on the body part from the raw thermal image data captured by the thermal camera 20. For more details on the thermal image processing system, see the following Figures 2 to 4 .

[0037] In operation, the user positions the thermal imager so that the body part to be imaged is within the field of view of the thermal camera. The output of the thermal camera can be displayed on a screen to assist in this operation.

[0038] Preferably, the thermal imager provides a real-time display of the scene to the user, and the imager is positioned by the user so that the object of interest (the foot) is located in the image. Preferably an alignment tool is used, such as a distance indicator / on screen overlay (similar to a visual camera system), etc. When the user is satisfied with the orientation, a button is pressed (or the screen is tapped) to capture the image.

[0039] Alternatively, an alignment guide may be projected from the device using an LED or similar, In such an arrangement, the user points the alignment guide to a specific feature of the target area of ​​the body part, or alternatively positions the body part / thermal imager so that the body part falls within the projected alignment guide.

[0040] A thermal image of the body part is then generated by the thermal image processing system 100 and written to the data repository 50. The image can be displayed on a screen, compared to a previous thermal image of the body part in the data repository 50, compared to an image of another foot, and / or used to trigger an alarm and / or highlight areas of potential concern exhibiting temperatures indicative of inflammation.

[0041] In the case where inflammation may occur on different parts / sides of a body part, the thermal imager 10 may include a capture protocol in the form of computer program instructions that are executed by the processor 40 and guide the user (e.g., through on-screen prompts on the display 60) to follow a predetermined measurement protocol to capture predetermined views of the body part. In the case of the foot, these views may include (in no particular order): left foot (plantar-plantar), right foot (plantar-plantar), left foot (superior-dorsal), right foot (superior-dorsal), left foot (lateral-lateral), right foot (lateral-lateral), left foot (medial-medial), right foot (medial-medial).

[0042] Preferably, the thermal imager is a microbolometer array operating at 8-14 μm. The microbolometer array is preferably a staring focal plane array, wherein its pixel elements are microbolometers sensitive to infrared radiation / temperature. In one embodiment, the size of the pixels is 17-25 μm. Every object above an absolute value of 0 emits infrared radiation. Infrared radiation has a strong correlation with temperature (T 4) relationship - so that its temperature can be determined by measuring the infrared radiation it emits. A thermal image is captured by measuring the infrared radiation received at each pixel of the array. The data can be encoded / compressed for storage efficiency (e.g., run length encoding) and can be processed using image processing in a manner similar to a pixel-based visual image.

[0043] It will be appreciated that the touch screen display may be replaced by another user interface such as a display and keyboard, or by other structures such as a Bluetooth connection to a smartphone or the like (which then acts as a remote user interface for controlling the thermal imager 10 and viewing images).

[0044] Thermal imager 10 may also include an I / O system for wired or wireless communication of acquired thermal imaging or other data. Battery power source 30 may be rechargeable and / or replaceable.

[0045] Figure 2 is a schematic diagram of a thermal image processing system for obtaining thermal image data of a body part according to an embodiment of the present invention.

[0046] The thermal image processing system 100 is configured to receive thermal image data 110 . The thermal image data includes thermal image data of a body part 111 and background thermal image data 112 .

[0047] To extract the thermal image data of the body part 111, the thermal image processing system 100 applies a classifier 120 to detect the contour of the body part. The classifier uses image data 130 other than the thermal image data 110 to perform the detection, as described below. Figure 3 and Figure 4 An example thereof is described.

[0048] Once the contour is detected, the thermal image processing system uses it to generate a mask 113. The thermal image processing system then applies the mask 113 to the thermal image data 110 to remove the background thermal image data 112 and obtain thermal image data of the body part 111.

[0049] Figure 3 is shown in more detail Figure 2 Schematic diagram of various aspects of the system.

[0050] In one embodiment, the thermal imager may further include a visual imaging sensor 70 configured to capture a spectrum of the imaged scene to generate visual image data (which may be stored separately from or together with the thermal image data 110). The visual imaging sensor 70 may, for example, capture an RGB image of the scene.

[0051] Figure 4 is shown in more detail Figure 2 Schematic diagram of various aspects of the system.

[0052] In a preferred embodiment, the thermal imager includes a stereoscopic visual imaging sensor 80 (in one embodiment, this is a pair of visual imaging sensors with a predetermined orientation and distance between them so that stereoscopic image processing can be performed).

[0053] The stereoscopic visual imaging sensor 80 is configured to capture a spectral stereoscopic image of the imaged scene to generate visual image data (which, as previously described, may be stored separately from the thermal image data 110, or may be stored together). The visual imaging sensor 80 may, for example, capture a pair of RGB images of the scene. In this embodiment, the classifier 120 uses the stereoscopic visual image data to detect the contour of the body part from the visual image data.

[0054] The classifier may optionally be configured to determine the spatial orientation of the body part based on the image data 130. For example, the classifier may use the stereoscopic image data to determine the spatial orientation of the body part, and the thermal image processing system may then modify the spatial orientation of the thermal image data obtained of the body part based on the determined spatial orientation to match the default spatial orientation.

[0055] In one embodiment, the thermal image processing system 100 can use stereoscopic image data to generate a 3D model (preferably a 3D topographical mesh) of the body part on which the thermal image data is registered. The 3D model can be generated using photometric stereo structure from motion or other methods.

[0056] Additional data can be overlaid onto the 3D model, such as interface pressure data, thermal measurements from thermal image data, such as maximum, minimum, mean, span, histograms, etc.

[0057] Figure 5 is shown in more detail Figure 2 Schematic diagram of other aspects of the system.

[0058] In this embodiment, the classifier 120 includes an artificial neural network that has been trained from image data 130 other than thermal image data including image data of a body part to detect the contour of the body part. In this embodiment, the image data other than thermal image data may or may not be image data of the body part of that particular person, and may include thermal image data, visual image data, or other image data (or a combination).

[0059] The neural network may be, for example, a convolutional neural network. The artificial neural network may be self-organizing and feed-forward.

[0060] It should be understood that certain embodiments of the present invention as described above can be incorporated as code (e.g., software algorithms or programs) residing in firmware and / or a computer-usable medium with control logic, which has control logic to enable execution on a computer system with a computer processor. Such a computer system typically includes a memory for providing output from code execution, according to which the processor is configured. The code can be built into firmware or software and can be organized into a set of modules, such as discrete code modules, function calls, procedure calls, or objects in an object-oriented programming environment. If implemented using modules, the code can include a single module or multiple modules that work in conjunction with each other.

[0061] Alternative embodiments of the present invention may be understood to include the parts, elements and features mentioned or indicated herein, individually or collectively, in any or all combinations of two or more parts, elements or features, wherein specific integers mentioned herein have known equivalents in the art to which the present invention relates, and such known equivalents are deemed to be incorporated herein as if individually set forth.

[0062] Although exemplary embodiments of the present invention have been described, it should be understood that those skilled in the art may make various changes, substitutions and alterations without departing from the invention as defined by the following claims and their equivalents.

Claims

1. A computer-implemented method for acquiring thermal image data of a body part, comprising: receiving image data of a body part, the image data comprising thermal image data and visual image data, the thermal image data comprising the thermal image data of the body part and background thermal image data, the visual image data being captured on the same scene as the thermal image data; performing a real-time scene display of the scene to guide a user to position a thermal camera for capturing the thermal image data so that the body part is within the image; applying a classifier to the image data to detect a contour of the body part from the visual image data; generating a mask according to the contour; as well as The mask is applied to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.

2. The method according to claim 1, wherein: The image data includes stereoscopic image data captured from the same scene as the thermal image data.

3. The method according to claim 1 or 2, wherein: The step of applying the classifier includes applying an artificial neural network trained from the image data of the body part to the received image data to detect the contour of the body part.

4. The method according to claim 2, further comprising: determining the spatial orientation of the body part from the stereoscopic image data by a classifier; The spatial orientation of the acquired thermal image data of the body part is modified according to the determined spatial orientation to match a default spatial orientation.

5. A thermal imager comprising: thermal cameras, visual imagers, displays, and processors; The thermal camera is used to capture thermal image data, wherein the thermal image data includes thermal image data of a body part and background thermal image data; The visual imager is used to capture visual image data on the same scene as the thermal image data; The display is used to display the scene captured by the thermal camera in real time to guide the user to position the thermal camera so that the body part is in the image; The processor is configured to execute computer program code to perform a classifier, including: computer program code for receiving image data of said body part, said image data comprising said thermal image data from said thermal camera; computer program code for applying a classifier to the image data to detect a contour of the body part from the visual image data; computer program code for generating a mask based on said contour; Computer program code for applying the mask to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.

6. The thermal imager according to claim 5, wherein: The visual imager is a stereoscopic visual imager for capturing stereoscopic visual image data from the same scene as the thermal image data.

7. The thermal imager according to claim 5, further comprising: Visual alignment tool to guide alignment of all body parts.

8. The thermal imager according to claim 7, wherein: The visual alignment tool includes a screen overlay on the display to guide alignment of the body part.

9. The thermal imager of claim 7, further configured to project a visual alignment guide from the device to be positioned at or around the body part.

10. The thermal imager according to claim 6, wherein: The processor is further configured to execute computer program code to determine the spatial orientation of the body part from the stereoscopic image data by a classifier, and to modify the spatial orientation of the obtained thermal image data of the body part according to the determined spatial orientation to match a default spatial orientation.

11. The thermal imager according to claim 6, wherein: The processor is also configured to generate a 3D model of the body part and project the thermal image data into the 3D model.

12. The thermal imager of claim 5, further comprising an artificial neural network trained from image data of a body part, the artificial neural network being configured to detect a contour of the body part from the thermal image data and to compare the contour with the contour generated by the processor.

13. The thermal imager according to claim 5, wherein: The classifier comprises an artificial neural network trained from image data of a body part, the artificial neural network being configured to detect a contour of the body part from the visual image data.

Images