System and method for acquiring thermal image data of body parts and thermal imager
Patent Information
- Application Number
- CN201980058841.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-07-16
- Filing Date
- 2019-07-16
- Publication Date
- 2025-01-24
- Estimated Expiration
- 2039-07-16
AI Technical Summary
Existing thermal imaging equipment suffers from size, cost and ease-of-use issues when detecting diabetic foot ulcers and is unable to provide high-precision temperature measurements, especially in large medical facilities or non-professional settings where existing equipment comes in contact with The measurement method will interfere with the accuracy of temperature measurement, and the device cannot accurately distinguish between body parts and background, resulting in uncertainty and error in temperature measurement.
Using a computer-implemented method, by receiving and processing thermal image data, using a classifier to detect the outline of body parts and generating a mask, combined with visual or stereoscopic vision image data, the background thermal image data is removed to obtain accurate thermal image data of body parts, Use neural network and machine learning technology for data processing to achieve non-contact thermal imaging.
Improves the accuracy and ease of use of thermal imaging equipment to effectively detect diabetic foot ulcers in non-professional settings, reduces uncertainty in temperature measurements, provides clinically relevant temperature data, and reduces equipment cost and size requirements , suitable for use by non-professional operators.
Smart Images
Figure CN112673394B8_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a system and method for acquiring thermal image data of body parts, and a thermal imager, particularly suitable for the application of identifying diabetic foot ulcers. Background Technology
[0002] People with diabetes are prone to severe foot ulcers, which can become infected and ultimately require amputation. Early detection is crucial for improving patient outcomes, allowing doctors to intervene before skin ruptures to protect the foot and administer antibiotics promptly if infection occurs.
[0003] It is estimated that up to 50% of people with diabetes experience foot neuropathy, which can lead to undetected, extensive skin damage, resulting in ulcers (diabetic foot ulcers (DFU)), infections, and amputations. With early preventative measures, the amputation rate is estimated to be reduced by 80%, and this cascade of events is largely avoidable. Mortality rates following DFU and amputation are high, with up to 70% of people dying within 5 years of amputation and approximately 50% dying within 5 years of developing diabetic foot ulcers. In addition to the adverse effects on patients, DFU and its complications are estimated to cost the National Health Service in England and Wales over £1 billion annually.
[0004] Medical research shows that skin temperature remains elevated by more than 2°C for up to 10 days before skin rupture occurs. Typically, skin temperature rises before any visible signs of infection or abnormality appear on the feet. This isn't necessarily related to infection; it could also be due to tissue damage caused by friction and the subsequent increased blood flow to the area.
[0005] Achieving the necessary accuracy in measurements and being able to monitor foot temperature over time presents a particular challenge.
[0006] Most doctors typically perform a visual inspection, tactile sensitivity test, and palpation of a diabetic patient's feet to detect changes in local temperature, which may indicate early lesions (pre-ulceration). These manual techniques are notoriously inaccurate and can be supplemented with thermal imaging (bringing the foot to a measuring device or platform for temperature measurement) or thermal imaging diagnostic techniques in areas with ulceration. However, off-the-shelf commercial thermal imaging and thermal imaging equipment and techniques also have a variety of drawbacks.
[0007] First, there is no known device that is suitable for use outside of large medical facilities or laboratories in terms of size, cost, and ease of use.
[0008] Ideally, such equipment should be available for use by podiatrists, nursing homes, etc., without requiring skilled operators.
[0009] Secondly, even with skilled operators, existing equipment cannot provide temperatures with clinically relevant uncertainties (i.e., better than ±0.3°C (k=2)). For example, many DFU current thermal imaging devices require the patient's foot to be in actual contact with the device. In this case (contact-based measurement), the temperature and thermophysical properties of the device can interfere with the temperature of the object (foot), thus affecting the accuracy of the measurement.
[0010] Even if other potential discrepancies and errors are minimized or eliminated, it has been determined that the accuracy of off-the-shelf commercial thermal imagers for absolute temperature measurements varies considerably. This is illustrated in a study ("Performance Testing of Thermal Imaging Systems to Evaluate Their Suitability for Quantitative Temperature Measurements," conducted by Whittam et al. at the 12th International Conference on Quantitative Infrared Thermal Imaging in Bordeaux, France, July 7-11, 2014"). https: / / www.ndt.net / search / docs.php3?showForm= off&id=17768 In this study, it was determined that 5 out of 6 thermal imaging devices had temperature measurement errors exceeding the manufacturer's specified error range at a distance of 1 meter from the target. Assuming the manufacturer's specified error range is typically ±2°C, and given that the thermal imaging devices cannot repeatedly and uniformly achieve this error range, they cannot provide clinically relevant temperatures (uncertainty < ±0.3°C). Therefore, it is highly questionable whether they can be relied upon to issue early warnings of persistent skin temperature increases >2°C.
[0011] Reliable comparisons of foot temperature changes over time require accurate foot spatial registration with each image acquisition. This can be challenging because operators (and their skills and methods) may differ, and foot position and size may vary between patients and measurements, thus distorting the accuracy of foot temperature measurements.
[0012] Therefore, there appears to be significant room for improvement in terms of equipment and methods. If the temperature elevation associated with neuropathy can be identified promptly and accurately, ulcers could be greatly reduced through preventative unloading, for example. Summary of the Invention
[0013] According to one aspect of the present invention, a computer-implemented method for acquiring thermal image data of body parts is provided, comprising:
[0014] Receive image data of a body part, the image data including thermal image data, the thermal image data including thermal image data of the body part and background thermal image data;
[0015] A classifier is applied to the image data to detect the contours of the body parts, and the classifier is configured to perform the detection using image data other than the thermal image data.
[0016] Generate a mask based on the contour; and
[0017] The mask is applied to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.
[0018] Embodiments of the present invention focus on a method and system in which supplementary data is used to guide the extraction of thermal images of body parts. The supplementary data can be other image data from the same scene (e.g., visual or stereoscopic images). The supplementary data may also (or optionally) include training data already used to train machine learning or artificial intelligence systems, such as neural networks, genetic algorithms, etc. For example, a classifier can be trained using a training library of foot images (visual, thermal, other image types, or some combination) to determine a contour from presented image data, which can then be used to generate a mask and extract thermal image data of the body part. In a preferred embodiment, the system is used in a handheld thermal imaging system to identify potential or predicted foot ulcers from captured thermal images.
[0019] In thermal imaging, it is generally assumed that the foreground object being imaged will be much warmer than its surroundings (background), thus background subtraction can be applied. However, this assumption is not always valid. There is no guarantee that the background temperature will differ significantly from the foreground body part, and it can vary depending on the time of day, season, or environment (e.g., this is not uncommon in hospitals and nursing homes, where embodiments of the invention can be used to provide comfort for people with limited mobility or who are bedridden). Furthermore, DFU is common around the big toe and sole of the foot. In cooler areas (e.g., due to circulation problems), there is a risk that the foot area may not be properly distinguished from the background. Embodiments of the invention attempt to address this problem by utilizing additional image data to more clearly distinguish the edges of body parts.
[0020] It should be understood that the methods can be implemented in parallel or in combination. For example, a neural network can generate what is considered a contour from thermal image data, in contrast to contours generated by a machine vision system from a stereo image of a scene. Alternatively, the neural network can operate on the stereo image and then apply the generated contours to the thermal image. Alternatively, the neural network can operate on the stereo image and then apply the generated contours to the thermal image.
[0021] Image data may include visual image data captured from the same scene as the thermal image data, and a classifier is configured to detect the contours of body parts from the visual image data.
[0022] Image data may include visual stereoscopic image data captured from the same scene as the thermal image data.
[0023] The classifier may include an artificial neural network or other artificial intelligence system trained from image data of body parts to detect the contours of body parts.
[0024] The classifier can be configured to determine the spatial orientation of the body part, and the method further includes the step of modifying the spatial orientation of the obtained thermal image data of the body part to match a default spatial orientation based on the determined spatial orientation.
[0025] While current focus is primarily on thermal imaging for detecting onset of foot neuropathy and triggering advanced preventative measures, the methods described in this application are also applicable to other body parts of human and / or animal subjects, such as the eyes, limbs, and other body parts.
[0026] The inventors of this application have conceived a non-contact thermal imaging device, system, and method that overcomes many of the shortcomings of known thermal imaging diagnostic devices and systems. Attached Figure Description
[0027] The embodiments of the present invention are described below by way of example only with reference to the accompanying drawings, wherein:
[0028] Figure 1 This is a schematic diagram of a thermal imager according to an embodiment of the present invention;
[0029] Figure 2 This is a schematic diagram of a thermal image processing system according to an embodiment of the present invention;
[0030] Figure 3 To explain in more detail Figure 2 A schematic diagram of various aspects of the system;
[0031] Figure 4 To explain in more detail Figure 2 A schematic diagram of various aspects of the system;
[0032] Figure 5 To explain in more detail Figure 2 A schematic diagram of optional aspects of the system. Detailed Implementation
[0033] Figure 1 This is a schematic diagram of a thermal imager according to an embodiment of the present invention.
[0034] The thermal imager 10 is preferably a handheld device, comprising a thermal camera 20, a battery power supply 30, a processor 40, a data storage device 50, and a touchscreen display 60.
[0035] The thermal imager also includes a thermal image processing system 100 executed by processor 40 to obtain thermal image data of body parts from raw thermal image data captured by thermal camera 20. Further details about the thermal image processing system are provided below. Figures 2-4 .
[0036] During operation, the user positions the thermal imager so that the body part to be imaged is within the camera's field of view. The camera's output can be displayed on a screen to assist in this process.
[0037] Preferably, the thermal imager provides a real-time scene display to the user, and the imager is positioned by the user so that the object of interest (the foot) is located in the image. Alignment tools are preferably used, such as a distance indicator / on a screen overlay (similar to a vision camera system). When the user is satisfied with the orientation, a button is pressed (or the screen is tapped) to capture the image.
[0038] Alternatively, an alignment guide can be projected from the device using an LED or similar device. In such an arrangement, the user points the alignment guide at a specific feature of a target area of the body part, or alternatively, positions the body part / thermal imager so that the body part falls within the projected alignment guide.
[0039] Then, the thermal image processing system 100 generates a thermal image of the body part and writes it to the data repository 50. This image can be displayed on a screen, compared with previous thermal images of the body part in the data repository 50, compared with an image of another foot, and / or used to trigger alarms and / or highlight areas of temperature indicating potential inflammation.
[0040] In cases where inflammation may occur in different parts / sides of a body part, the thermal imager 10 may include a capture protocol in the form of computer program instructions, which is executed by the processor 40 and instructs the user (e.g., via on-screen prompts on the display 60) to follow a predetermined measurement protocol to capture predetermined views of the body part. If it is a foot, these views may include (in no particular order): left foot (sole-to-foot), right foot (sole-to-foot), left foot (upper foot-back), right foot (upper foot-back), left foot (outer side-lateral), right foot (outer side-lateral), left foot (inner side-middle), right foot (inner side-middle).
[0041] Preferably, the thermal imager is a micro-bolometer array operating at 8–14 μm. The micro-bolometer array is preferably a staring focal plane array, wherein its pixel elements are micro-bolometers sensitive to infrared radiation / temperature. In one embodiment, the pixel size is 17–25 μm. Every object above absolute zero emits infrared radiation. Infrared radiation has a strong (T0)-temperature relationship. 4This relationship allows us to determine the temperature of an object by measuring the infrared radiation it emits. Thermal images are captured by measuring the infrared radiation received at each pixel of the array. The data can be encoded / compressed to improve storage efficiency (e.g., run-length encoding) and can be processed using image processing in a manner similar to pixel-based visual images.
[0042] It will be understood that the touchscreen display can be replaced by another user interface such as a monitor and keyboard, or by other structures such as a Bluetooth connection with a smartphone (which then acts as a remote user interface for controlling the thermal imaging system 10 and viewing images).
[0043] The thermal imager 10 may also include an I / O system for wired or wireless communication of the acquired thermal images or other data. The battery power supply 30 may be rechargeable and / or replaceable.
[0044] Figure 2 This is a schematic diagram of a thermal image processing system for obtaining thermal image data of body parts according to an embodiment of the present invention.
[0045] The thermal image processing system 100 is configured to receive thermal image data 110. The thermal image data includes thermal image data of body parts 111 and background thermal image data 112.
[0046] To extract thermal image data of body part 111, thermal image processing system 100 applies classifier 120 to detect the contour of the body part. The classifier uses image data 130 other than thermal image data 110 to perform the detection, as described below. Figure 3 and Figure 4 Describe an example.
[0047] Once the contour has been detected, the thermal image processing system uses it to generate a mask 113. The thermal image processing system then applies the mask 113 to the thermal image data 110 to remove the background thermal image data 112 and obtain the thermal image data of the body part 111.
[0048] Figure 3 To show in more detail Figure 2 A schematic diagram of various aspects of the system.
[0049] In one embodiment, the thermal imager may further include a visual imaging sensor 70 configured to capture the spectrum of the imaging scene to generate visual image data 130a (which may be stored separately from or together with the thermal image data 110). The visual imaging sensor 70 may, for example, capture an RGB image of the scene.
[0050] Figure 4 To show in more detail Figure 2 A schematic diagram of various aspects of the system.
[0051] In a preferred embodiment, the thermal imager includes a stereo vision imaging sensor 80 (in one embodiment, this is a pair of vision imaging sensors with a predetermined orientation and distance between them, thereby enabling stereo image processing).
[0052] The stereo vision imaging sensor 80 is configured to capture a stereo image of the spectrum of the imaging scene to generate visual image data 130b (which, as previously described, may be stored separately from or together with the thermal image data 110). The visual imaging sensor 80 may, for example, capture a pair of RGB images of the scene. In this embodiment, the classifier 120 uses the stereo vision image data 130b to detect the contours of body parts from the visual image data.
[0053] The classifier can optionally be configured to determine the spatial orientation of a body part based on the image data 130. For example, the classifier can use stereo vision image data to determine the spatial orientation of the body part, and then the thermal image processing system can modify the spatial orientation of the obtained thermal image data of the body part according to the determined spatial orientation to match the default spatial orientation.
[0054] In one embodiment, the thermal image processing system 100 can use stereoscopic image data 130b to generate a 3D model (preferably a 3D topographic mesh) of a body part on which thermal imaging data is registered. The 3D model can be generated using photometric stereo structures derived from motion or other methods.
[0055] Other data can be overlaid on the 3D model, such as interface pressure data, thermal measurements from thermal imaging data, such as maximum, minimum, mean, span, histogram, etc.
[0056] Figure 5 To show in more detail Figure 2 A schematic diagram of other aspects of the system.
[0057] In this embodiment, classifier 120 includes an artificial neural network trained from image data 130 other than thermal image data including image data of body parts, for detecting the contours of body parts. In this embodiment, the image data other than thermal image data may or may not be image data of body parts of that particular person, and may include thermal image data, visual image data, or other image data (or combinations thereof).
[0058] This neural network can be, for example, a convolutional neural network. Artificial neural networks can be self-organizing and feedforward.
[0059] It should be understood that certain embodiments of the invention described above can be incorporated as code (e.g., software algorithms or programs) residing in firmware and / or a computer-usable medium with control logic to enable execution on a computer system with a computer processor. Such a computer system typically includes memory for providing output from code execution, configuring the processor accordingly. The code can be constructed as firmware or software and can be organized as a set of modules, such as discrete code modules, function calls, procedure calls, or objects in an object-oriented programming environment. If implemented using modules, the code can comprise a single module or multiple modules that work together.
[0060] The alternative embodiments of the invention may be understood to include the components, elements and features mentioned or indicated herein, individually or collectively, in any combination or all combinations of two or more components, elements or features, wherein specific integers mentioned herein have known equivalents in the field to which the invention relates, and such known equivalents are considered to be incorporated herein as if set forth separately.
[0061] Although exemplary embodiments of the invention have been described, it should be understood that various changes, substitutions and modifications can be made by those skilled in the art without departing from the invention as defined in the following claims and its equivalents.
Claims
1. A computer-based method for acquiring thermal image data of body parts, comprising: Receive image data of a body part, the image data including thermal image data, the thermal image data including thermal image data of the body part and background thermal image data; A classifier is applied to the image data to detect the contours of the body parts, and the classifier is configured to perform the detection using image data other than the thermal image data. Generate a mask based on the described contour; as well as The mask is applied to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.
2. The method according to claim 1, wherein, The image data includes visual image data captured from the same scene as the thermal image data, and the classifier detects the contours of the body parts from the visual image data.
3. The method according to claim 2, wherein, The image data includes visual stereoscopic image data captured from the same scene as the thermal image data.
4. The method according to claim 1, 2 or 3, wherein, The steps of applying the classifier include: applying an artificial neural network trained from image data of a body part to the received image data to detect the contour of the body part.
5. The method according to claim 4, further comprising: The spatial orientation of the body parts is determined by a classifier; The spatial orientation of the obtained thermal image data of the body part is modified according to the determined spatial orientation to match the default spatial orientation.
6. A thermal imager, comprising: Thermal camera and processor; The processor is configured to execute computer program code to perform the classifier, including: Computer program code for receiving image data of body parts, the image data including thermal image data from the thermal camera, the thermal image data including thermal image data of the body parts and background thermal image data; Computer program code for applying a classifier to the image data to detect the contours of the body parts, the classifier being configured to perform the detection using image data other than the thermal image data; Computer program code for generating a mask based on the contour; Computer program code for applying the mask to the thermal image data to remove the background thermal image data and obtain thermal image data of the body part.
7. The thermal imager of claim 6 further includes a visual imager for capturing visual image data in the same scene as the thermal image data, wherein the computer program code is used to apply the classifier to detect the contour of the body part from the visual image data.
8. The thermal imager according to claim 7, wherein, The visual imaging device is a stereo vision imaging device.
9. The thermal imager according to claim 6, 7 or 8, wherein, The computer program code for applying the classifier includes: computer program code for applying an artificial neural network trained from image data of a body part to the received image data to detect the contour of the body part.
10. The thermal imager according to claim 9, wherein, The processor is further configured to execute computer program code to determine the spatial orientation of the body part through a classifier, and to modify the spatial orientation of the obtained thermal image data of the body part according to the determined spatial orientation to match the default spatial orientation.