Thermal imaging systems and methods

Abstract

A thermal imaging system configured for use with a respirator mask is presented that include an in-mask display unit configured to be mounted within an interior volume of a respirator mask and including an in-mask micro-display. The system also includes a wearable thermal imaging camera that is configured to capture a thermal image and to communicate at least a portion of the captured thermal image to the in-mask micro-display. The system also includes an image processing system including a processor and memory, the system being configured to: detect an object in the thermal image, generate a callout to indicate the detected object, and compile a composite image, the composite image includes the thermal image and the generated callout.

Description

[0001] THERMAL IMAGING SYSTEMS AND METHODS

[0002] Background

[0003] Thermal imaging cameras have found increasing use in firefighting, search and rescue, and the like. Such cameras capture infrared light and transform it into a visible-light display, so that, for example, hot spots in a building may be detected.

[0004] Summary

[0005] In broad summary, herein is disclosed a thermal imaging system configured for use with a respirator mask that include an in-mask display unit configured to be mounted within an interior volume of a respirator mask and including an in-mask micro-display. The system also includes a wearable thermal imaging camera that is configured to capture a thermal image and to communicate at least a portion of the captured thermal image to the in-mask micro-display. The system also includes an image processing system including a processor and memory, the system being configured to: detect an object in the thermal image, generate a callout to indicate the detected object, and compile a composite image, the composite image includes the thermal image and the generated callout.

[0006] These and other aspects will be apparent from the detailed description below. In no event, however, should this broad summary be construed to limit the claimable subject matter, whether such subject matter is presented in claims in the application as initially filed or in claims that are amended or otherwise presented in prosecution.

[0007] Brief Description of the Drawings

[0008] FIG. 1 illustrates an example environment in which systems and methods herein may be useful.

[0009] FIGS. 2A and 2B illustrate an exemplary respirator mask assembly and an in-mask display unit of the respirator mask in accordance with embodiments herein.

[0010] FIG. 3 illustrates a method of providing an object recognition overlay to an in-mask display unit in accordance with embodiments herein.

[0011] FIGS. 4-8B illustrate example user interfaces that may be presented to a user in accordance with embodiments herein.

[0012] FIG. 9 illustrates a schematic of a user interface that may be presented to a user in accordance with embodiments herein.

[0013] FIG. 10 illustrates a scene analyzing system in accordance with embodiments herein.

[0014] FIG. 11 is a block diagram of an image processing system architecture. FIGS. 12-13 illustrate example mobile computing devices that can be used in embodiments shown in previous Figures.

[0015] FIGS. 14-15 are block diagrams of a computing environment that can be used in embodiments shown in previous Figures.

[0016] Like reference numbers in the various figures indicate like elements. Some elements may be present in identical or equivalent multiples; in such cases only one or more representative elements may be designated by a reference number but it will be understood that such reference numbers apply to all such identical elements. Unless otherwise indicated, all figures and drawings are not to scale and are chosen for the purpose of illustrating different embodiments of the invention. In particular the dimensions of the various components are depicted in illustrative terms only, and no relationship between the dimensions of the various components should be inferred from the drawings, unless so indicated.

[0017] Detailed Description

[0018] Although terms such as “first” and “second” may be used in this disclosure, it should be understood that those terms are used in their relative sense only unless otherwise noted. As used herein as a modifier to a property or attribute, the term “generally”, unless otherwise specifically defined, means that the property or attribute would be readily recognizable by a person of ordinary skill but without requiring a high degree of approximation (e.g., within + / - 20 % for quantifiable properties). The term “substantially”, unless otherwise specifically defined, means to a high degree of approximation (e.g., within + / - 10% for quantifiable properties). The term “essentially” means to a very high degree of approximation (e.g., within plus or minus 2 % for quantifiable properties); it will be understood that the phrase “at least essentially” subsumes the specific case of an “exact” match. However, even an “exact” match, or any other characterization using terms such as e.g. same, equal, identical, uniform, constant, and the like, will be understood to be within the usual tolerances or measuring error applicable to the particular circumstance rather than requiring absolute precision or a perfect match.

[0019] As used herein, the vertical axis and upward and downward directions along the vertical axis have their ordinary meaning with respect to a respirator mask (and associated equipment) worn by an upright human user. Similarly, the lateral axis and right and left directions along the lateral axis have their usual meanings with reference to a respirator mask worn by an upright human user. Terms such as forward and rearward as used with respect to a respirator mask and components thereof, respectively imply directions away from, and toward, the face of the user. The above terminology also applies to images shown on a display of a thermal imaging system, as discussed in detail herein. It will be understood that such terminology is used for convenience of description, noting that in actual practice, a person wearing a respirator mask and / or using a thermal imaging system associated with the respirator mask, may occasionally assume an orientation other than standing strictly upright.

[0020] The term field of view has its usual optical meaning as quantified in terms of a solid angle. By a thermal imaging camera with a wide-angle field of view is meant that the camera captures thermal images at a field of view of at least 65 degrees; by a wide-angle thermal image is meant such an image captured at a field of view of at least 65 degrees. The term micro-display denotes a display screen with an active display area that has a diagonal length of less than 2.0 inches. The term “configured to” and like terms is at least as restrictive as the term “adapted to,” and requires actual design intention to perform the specified function rather than mere capability of performing such a function. All references herein to numerical values (e.g. dimensions, ratios, and so on), unless otherwise noted, are understood to be calculable as average values derived from an appropriate number of measurements.

[0021] Disclosed herein are object detection and recognition systems that may be used in conjunction with a thermal imaging device. For example, embodiments herein may include a thermal imaging device that captures images for display on a display component. Object detection and recognition systems herein may analyze and identify objects in the captured images in some embodiments. In some embodiments, systems and methods herein may, based on identified objects, modify or augment an image prior to display on a display component. For example, debris or objects detected in a pathway of an individual may be highlighted, outlined, or otherwise indicated. A door handle too hot to touch may be indicated, in another example. Flammable, explosive or other hazardous chemicals may trigger an alert to evacuate or other safety precautions that must be taken.

[0022] FIG. 1 illustrates an example environment in which systems and methods herein may be particularly useful. A number of firefighters 130 may arrive at a location 100 where an active fire 140 is in progress. Firefighters 130 may enter an environment 140 where visibility is low - fires generate a lot of smoke making it difficult for an individual to see or orient themselves. A supervisor or site manager 120 may be responsible for dispatching or assigning individual firefighters 130 to different tasks.

[0023] When an emergency worker 30 arrives on scene 100, a layout of the hazardous area 140 is generally not available. It is common for workers 130 to experience disorientation due to the low visibility and lack of information when entering a scene (e.g. to fight the fire, rescue a victim or fellow team member, etc.). It can be difficult to conduct a rescue operation under such conditions. The difficulty is further complicated when debris or objects are in the path of a rescue worker. For example, a fire in a gymnasium may present additional tripping hazards to rescue workers. Additionally, in an office environment there may be obstructions that render a location or corridor impassable.

[0024] Some potential rescue technology has been developed that can assist search and rescue teams. For example, radio waves emanating from a downed, trapped or lost personnel may be received by another rescuer and used for location purposes. However, if a door or wall is between the downed individual and the rescuer, these are of limited value. Published PCT Application WO 2021 / 255592, published December 12, 2021, describes use of fiducial markers, which emit wireless signals detectable by a sensor, to identify locations. Fiducial markers may be particularly useful for documenting an ingress path. However, their use for leaving an area is limited to scenarios where no new debris or obstructions are formed, require a rescuer to have the fiducial markers on hand and remember to place them periodically. The sensor system for detecting said fiducial markers must also be present and functional.

[0025] Thermal imaging systems are often used to provide additional information for rescue workers. Thermal imagers function by detecting radiation from heat sources and the ambient background. If there is sufficient differences in the apparent temperature between heat sources and heat sinks, then quality imagery can be obtained. However, as illustrated in FIGS. 4-9, the imagery from a thermal imaging device, while greatly improved over human vision alone, can still be difficult for an individual to interpret. And, in environments where speed saves lives, any additional information that assists in quick understanding of a scene is beneficial.

[0026] Systems and methods herein may detect and identify objects in a scene, based on captured thermal imagery. Systems and methods herein may modify a real-time feed of thermal imagery provided to a rescue worker, for example using an in-mask display unit. Systems and methods herein may also provide information, alerts, or a modified / augmented image feed to a supervisor or site manager 20.

[0027] As used herein, the term “real-time” refers to data is processed within milliseconds so that it is available virtually immediately as feedback. While some delay due to processing are inevitable, “realtime” is intended to cover systems and methods where data can be collected or entered and a user can then interact with it without noticeable delay. E.g. a user may make a data entry into a system, and the data entry is then substantially immediately available for viewing or editing.

[0028] FIGS. 2A and 2B illustrate an exemplary respirator mask assembly and an in-mask display unit of the respirator mask in accordance with embodiments herein. It is noted that FIGS. 2A-2B illustrate one example of a respirator mask assembly, and embodiments described herein may be implemented in other respirator mask assembly designs.

[0029] FIG. 2A illustrates an exemplary respirator mask assembly 210, comprising a respirator mask 214 and various components thereof and equipment associated therewith, e.g. a head harness (unnumbered) to hold the mask in place on the user’s face, a generally forward-facing lens 211 through which the user can see, and so on. Such a respirator mask 214 will define an interior volume 216 when fitted to the face of a human user. A chief purpose of the respirator mask will be to deliver breathable air to the user. In aid of this, such a mask will often comprise a facepiece 212 that resides within interior volume 216 that fits about the nose and mouth of the user to deliver breathable air thereto. A full facepiece seals along the outer perimeter to portions of the forehead, temples, cheeks and chins.

[0030] The respirator mask may comprise various connections, fittings and the like by which breathable air can be delivered to the user; typically these fittings are in communication with facepiece 212 so that air can be delivered to the user via the facepiece. In some embodiments the breathable air may be provided (via one or more regulators 208) from one or more pressurized air tanks (not shown). In some embodiments the breathable air may be filtered air, having passed through, for example, one or more filter cartridges mounted onto fittings of the respirator mask.

[0031] Thermal imaging system 201 will rely on at least one thermal imaging camera 218. In FIG. 2A, thermal imaging camera 218 is mounted on the outside of assembly 210. However, in some arrangements, the thermal imaging camera can be mounted in another position on assembly 210, or on some other item worn by the user of the respirator mask, for example a helmet or a garment (such as a belt, a protective article of clothing such as a coat, etc.). As disclosed herein, any such thermal imaging camera, whether mounted on the respirator mask or on a helmet or garment, will be termed a “wearable” camera, and allows hands-free use of the camera. A wearable camera will be distinguished from, e.g., a hand-held camera. Regardless of the exact item on which the wearable thermal imaging camera is mounted, the camera 18 will typically face generally forward. In some embodiments, the position and / or orientation of the camera may be adjustable (e.g. to take into account the height of a user of the respirator mask).

[0032] The thermal imaging system 201 may provide an image feed for display on an in-mask display unit 220 (shown in FIG. 2B) that is positioned within the interior volume 216 of the respirator mask 214. In the exemplary arrangement depicted in FIGS. 2A-2B, the in-mask display unit comprises an in-mask micro-display 230 that is mounted in a generally monocular eyepiece 217 (e.g. in the general form of a tapered tube). The in-mask display unit 220 may further comprise at least one lens (often referred to in the vernacular as an “optic”) mounted somewhere in the eyepiece, rearward of the microdisplay so as to be between the in-mask micro-display and the user’s eye. In many embodiments the display unit may comprise (at least) two lenses, e .g . an ocular lens 231 that is positioned at the rearward end of the eyepiece 217, and an objective lens 32 positioned between the ocular lens and the microdisplay. Other configurations are also expressly contemplated for other embodiments herein.

[0033] Eyepiece 217 of in-mask display unit 220 can be attached to a support chassis 215, which in turn in turn can be attached to facepiece 212 and / or to any other component of respirator mask 214. In the depicted embodiment, chassis 215 is attached to an assembly (unnumbered) that is itself attached to facepiece 212 and / or to some other component of mask 214. In general, any suitable method or mechanism can be used to position the in-mask display unit 220, and the in-mask micro-display 230 thereof, within interior volume 216 of respirator mask 214. In some embodiments, the position and / or orientation of eyepiece 217 can be adjusted e.g. by way of various hinged connections or other suitable mechanisms.

[0034] In-mask display unit 220 is positioned and sized so as to not unduly interfere with the user’s ability to see out through the front lens 211 of the respirator mask 214. Such considerations also often preclude using an in-mask display unit that is large enough to be seen with both eyes. Thus, in many embodiments the in-mask display unit 220 comprises a monocular design in which a single eyepiece 17 is provided in which in-mask micro-display 230 is positioned so that it is only visible to a single eye.

[0035] Furthermore, the size of the in-mask micro-display 30 itself will necessarily be limited, e.g. to a diagonal length of less than 2.0 inch, in comparison to displays of 3.5 inch diagonal length as are commonplace on many hand-held thermal imaging devices. A display that is 2.0 inch or less in diagonal length will be termed a micro-display as used herein. In some embodiments, such a microdisplay may be even smaller, e.g. with a diagonal length of less than or equal to 1.5, 1.0, 0.80, 0.70, 0.60, 0.50, or 0.40 inches. (These dimensions refer to the active display area that presents an image; the active display area of a micro-display is often surrounded by a picture-frame border that contains various electronic equipment used to drive the display and to address the pixels of the display.) Such a micro-display may rely on any appropriate technology and operating mechanism, e.g. LCD, AMLCD, OLED, AMOLED, FLCOS, and so on.

[0036] The wearable thermal imaging camera 218 is configured to capture a thermal image, at least a portion of which is communicated to the in-mask micro-display 230. By this is meant that the captured image 250 will be transformed into data by a control system 219 with the data being sent to microdisplay 230 and reconstituted as a displayed image 260. It is noted in passing that discussions herein are phrased primarily in terms of capturing an image, communicating an image to the in-mask microdisplay 230 of the in-mask display unit 220, and so on. This terminology is used for convenience of description; in actuality, the thermal imaging system will capture, process, communicate, and display a continuing stream of images e.g. at a suitable frame rate. In some embodiments, such communication can occur through a “wired” (physical) connection, e.g. a wire or optical fiber. However, in some embodiments, the communication can be wireless, e.g. by way of Bluetooth® or any other suitable radiofrequency communication protocol.

[0037] The communication of images from camera 218 to micro-display 230, as well as other aspects of the operation of the thermal imaging system, will be governed by a control system 219 (indicated in generic representation in FIG. 2B). Such a control system can comprise any suitable hardware, e.g. one or more computing devices along with whatever ancillary items and components are needed to operate the computing device(s). Such a computing device will typically comprise one or more processors as present e.g. in an integrated circuit supported on a printed circuit board or flex circuit. In various embodiments, some or all of the components of control system 219 can be located, and operation of these components can occur, in any suitable location, e.g. in the cloud. However, it may be particularly convenient that all components of control system 19 will be locally located in or on respirator mask 214, e.g. in an electronics module provided forthat purpose.

[0038] Control system 219 can comprise any components and / or functionalities (e.g. data-storage memory, wireless transmitters and receivers, and so on) as needed in order to operate camera 218 and micro-display 230, to process the images captured by camera 218 and communicate them to microdisplay 230, to interact with a user of the thermal imaging system, and so on. Other electronic components (e.g. an accelerometer 221 and / or a microphone 222) may be present in or on respirator mask 214 and may also be operated by control system 219. Control system 219 may comprise any suitable items as needed, e.g. power switches, one or more other mechanical or virtual switches or controls to allow the user to govern various aspects of the operation of the thermal imaging system (e.g. the sensitivity level of the thermal imaging camera and the brightness level of the micro-display), and so on. Various other items may be present. For example, a subsidiary processor and / or circuit board may be present whose function is to operate micro-display 230; camera 218 may similarly comprise a subsidiary circuit board, processor, or the like, whose function is to operate camera 218. Control system 219, camera 218, and micro-display 230 may all be provided with electrical power from a suitable power source. In various embodiments, a single power source may be used to power all such items; however, in some embodiments one or more of these items may be provided with an individual power source, e.g. a battery.

[0039] The above is not meant to be an exhaustive description of all of the various items, components, and functionalities that may be present in a thermal imaging system or in a thermal imaging camera or a micro-display thereof. Thermal imaging systems, in particular systems with in-mask display units as may find use with respirator masks, are described in detail in U. S. Patent Application Publication 2019 / 0118008; and, in U.S. Provisional Patent Application 62 / 745154 and in the resulting PCT Application published as WO 2020 / 075045, all of which are incorporated by reference in their entirety herein.

[0040] Additionally, while the arrangements disclosed in FIGS. 2A-2B describe a single thermal imaging device, it is expressly contemplated that some embodiments include additional thermal imaging cameras. E.g., one or more additional thermal imaging cameras may be provided e.g. on the rear brim of a protective helmet; or, on the front crown, or the rear crown, of the helmet. Thus, in some such embodiments a primary subimage may be obtained from a first, generally forward-facing thermal imaging camera; and, a secondary subimage may be obtained from a second thermal imaging camera that may face e.g. generally rearward, upward, rearward / upward, to one lateral side or another, and so on. It is expressly contemplated that, in some embodiments, a secondary subimage does not come from the same captured image as a primary subimage. It is also noted that in general, a secondary subimage that is shown in auxiliary view does not necessarily have to include the entirety of the portion (or portions) of a captured thermal image that was not shown in default view. However, in some embodiments it may do so.

[0041] Although not discussed previously herein, it is noted that if desired, in some embodiments the system may be configured so that a primary subimage and / or a secondary subimage may be zoomed in (magnified) or zoomed out (shrunk). Typically, any such zooming in or out will be performed by way of the control system manipulating the stream of images rather than by way of e.g. mechanically changing the focal length of the thermal imaging camera. In some embodiments any such operation may be independent of, and controlled independently of, the above-disclosed arrangements, e.g. by voice command. However, if desired, in some embodiments such zooming may occur in combination with the herein-disclosed arrangements. For example, the system may be configured so that when a secondary subimage is displayed that comprises e.g. only a single, rather small, hot spot, the system may automatically zoom in so the portion of the subimage that includes the hot spot is magnified. Again, it will be appreciated that the thermal imaging system may offer many of these modes and functionalities as options that can be chosen between, depending e.g. on circumstances and user preference. Although it may typically not be preferred, the system may be configured to allow the user to specify that the entirety of the captured thermal image is shown on the micro-display for at least some prechosen period of time. That is, even with the loss of detail that may result from the compression of the entire captured thermal image to fit on the micro-display, the user may still desire to see the entire image for some particular purpose, on occasion.

[0042] Thermal images (i.e., infrared (IR) thermal images) differ from visible light images. More specifically, thermal-imaging devices sense / capture invisible heat radiated from objects and do not require light (as visible light imaging devices) to capture images. That is, thermal imaging devices may be used in any condition of lighting. Further, thermal imaging may be used in industrial environments and / or by first responders and / or providers of emergency services. Thermal imaging devices may be used by firefighters at least because the imager, e.g., operating in a range of 8 to 14 micron, is generally insensitive to smoky conditions associated with fire that would otherwise obscure the image. The image may be provided in shades of white, grey, and black. The image may be colorized to highlight environmental conditions such as “hot spots.” Further, thermal imaging may be integrated with edge detection processes / algorithms to further provide details about objects in the image.

[0043] Thermal imagery, and in-mask display systems like that discussed in FIGS. 2A-2B have greatly improved the ability of emergency response workers to see and navigate in low visibility environments. However, the ability of the human brain to process imagery, in real-time in a stressful environment, is limited. Firefighters, for example, are trained to operate in black-out conditions and “feel” their way around a fire scene. The focus is completing an objective while maintaining breathable air. Limited focus is available for viewing and interpreting shapes on a heads-up display.

[0044] Systems and methods in embodiments herein improve a user’s ability to detect obstacles, hazards and rescue objectives. For example, in combination with thermal imagery, sensors (such as altimeters, radar, compass, and / or cameras) can determine where a fire is as well as where hot and cold spots are. Such information may be particularly useful in guiding an individual within a scene. Identifying doors and windows, as well as knob, latch or handles, in context with thermal imagery, provides information that could be critical in making a “GO” or “NO GO” decision. Systems and methods herein may be able to identify rescue objectives - e.g. where a human or pet victim is located - using thermal imagery and object recognition. Object may be more easily avoided - such as exercise equipment, furniture, etc. Systems and methods herein may be able to identify potentially dangerous hazards, such as gas can, firearms or ammunition, as well as hazard indicators such as OSHA symbols. Systems and methods herein may also identify objects that may be helpful for a user, such as stand pipes, hose packs, extinguishers, exit signs, doors and windows.

[0045] Thermal imagery may provide additional information to identified objects. For example, a live human victim will have a different heat profile than a mannequin or deceased individual.

[0046] FIGS. 2A-2B illustrate one example in-mask display system. However, it is expressly contemplated that other display configurations are possible, such as augmented-reality enabled eyewear or an auxiliary display, such as the monocle described and illustrated in PCT Publication WO 2022 / 123374, published June 6, 2022. Other configurations are expressly contemplated.

[0047] FIG. 3 illustrates a method of providing an object recognition overlay to an in-mask display unit in accordance with embodiments herein. While method 300 describes providing an overlay that may appear over an image or video feed provided in real-time to a display visible to a user, it is expressly contemplated that other methods of communicating information to a user are possible. For example, object identification may be provided separately from a video feed, may be provided through another feedback mechanism such as audio or haptic feedback, or in another suitable manner.

[0048] In block 310, an image of a scene is received. The image may be received from an image capture device coupled to an article of PPE, such as a helmet, self-contained breathing apparatus (SCBA), purified air powered respirator (PAPR), or another suitable item, such that the image capturing device moves with the user and captures an image that approximates, or is close to, the scene as viewed by a user. However, the image capturing device may be positioned elsewhere, e.g. coupled to a user’s body through an article of clothing.

[0049] The image may be captured using a thermal imager 304, such that the image includes data regarding the relative temperature of different parts of the scene. However, other imagers may be used, as indicated in block 306, such as a camera, an infrared camera, etc. Additionally, it is expressly contemplated that sensor data from one or more sensors 308 is received. Sensors 308 may include global positioning systems, altimeters, accelerometers, or other inertial measurement devices, microphones LIDAR, radar, gyroscopes, and / or fiducial markers as described in WO 2021 / 255592, published December 12, 2021, or another suitable sensor. However, it is expressly contemplated that other sensor information may be received from sensors associated with a particular user, sensors associated with another user, environmental sensors, etc.

[0050] In block 320, the received image is analyzed, for example by a scene analyzing system. The analysis may include other sensor data received simultaneously with the image data or sensor data received separately from the image data. Image analysis may include applying one or more machine learning algorithms to the image data. For example, one or more edge detection algorithms 312 may be applied, for example as described in FIG. 3 and the accompanying description of PCT Application PCT / IB2023 / 052654, filed March 17, 2023, which is herein incorporated by reference. Edge detection, as used herein, includes detecting at least one edge. In some embodiments, edge detection includes determining an edge arrangement (e.g., edge orientation forming an object shape, edge orientation in space) of the at least one detected edge of the at least one object. In some embodiments herein, edge detection also includes assigning the edge arrangement to at least one of at least one object category (e.g., window, door, sign, etc.) and a structure category (e.g., a warehouse, industrial building, chemical plant, power plant, etc.)

[0051] Edge detection processes / algorithms may be tuned using parameters associated with image processing. In addition, thermal imaging integrated with edge detection of objects may be beneficial at least because thermal conditions may be correlated to certain areas of a structure where a fire occurs. For example, a high temperature gas layer at a ceiling of the structure where the fire is present may be shown in a composite image that includes a base image with thermal indicators (e.g., coloring of the image indicating the high temperature of the gas) and edges laid over the base image indicating the edges of the ceiling. In addition, thermal imaging integrated with edge detection may be beneficial at least because a composite image may show different hazards associated with different conditions of objects such as doors that are open, confined stairways, hot doors, etc. Characteristics of objects such as transparency (e.g., of a pane of glass) that are detected by thermal imaging (and undetected by visual light imaging) may be further enhanced by overlaying detected edges on the object on the thermal image.

[0052] Edge detection and thermal imaging may also allow use of lower resolution camera cores which may be low power consumption camera cores. Low power consumption is important in firefighting as firefighters using thermal imaging may be without access to a charging source for long periods of time. In addition, lower power consumption of thermal imaging processes is also important for other users, such as users of thermal imaging devices in industrial environments, pipeline operations, HAZMAT operations, storage / warehousing, in remote locations and / or locations where a power source is not available. Therefore, a combination edge detection processes to analyze thermal images efficiently (i.e., with low power consumption) while still providing information that is relevant to end users is described.

[0053] In some embodiments, objects (e.g., on a thermal image) are identified and / or displayed, e.g., so that the end user can make informed decisions about a next step to be taken for a mission. In some other embodiments, visual information (e.g., associated with a composite image including a thermal image) may be accompanied by audio related information in a Bone Conduction Headset (BCH) system for products so equipped.

[0054] In some embodiments, several edge detection processes may be performed, where one the edge detection process may provide a better visual result than another. Different results may be due to environmental conditions (e.g., smoke, water fog, obstacle congestion) and / or other factors (e.g., present during machine learning of collected data). Multiple edge detection processes may be tested in a predetermined environment, and the edge detection process that meets a predetermined quality parameters (e.g., clearest image) may be selected. A number of provisionally assigned images may be maximized. The provisionally assigned imaged with the most assigned images (i.e., the fewest ambiguous images) may be selected. Any edge detection process may be selectable by the user, e.g., the user being able to toggle edge detection processes based on a preferred view.

[0055] As indicated in block 314, a machine learning based object recognition algorithm may be applied to the scene in some embodiments. The object recognition algorithm may identify objects in the scene using the edge detection results, in some embodiments. However, it is expressly contemplated that the object recognition algorithm may identify objects using other suitable methods. For example, a machine learning algorithm may be trained on a dataset including a number of images of known objects captured in low-visibility conditions and / or using a thermal imager.

[0056] In some embodiments, the object recognition algorithm may identify an object based on a cue. For example, window glass reacts to infrared light in a similar manner to a mirror. A separate cue, such as identification of a window pane and / or sashes may cause the algorithm to categorize an item as a window; while the presence of a countertop may cause the algorithm to categorize an item as a mirror.

[0057] As an example, an edge detection algorithm may detect parallel lines on a floor that are about three inches apart. This may be identified by the object recognition algorithm as a fire hose, which can serve as a lifeline for a firefighter.

[0058] As another example, a ceiling may be identified based detected edges of a wall with the ceiling. Could be the floor as well. Look up at the ceiling (e.g. in a warehouse - if its getting hot near the ceiling there could be a roof collapse, may trigger evac). Detecting airflow and making decisions about the airflow - any thermal imaging class (Jason Patterson) can detect smoke billowing / fast flow of air outward.

[0059] If we can detect floor (as user moves) / wall intersection AND ceiling / wall; can determine the heat layers (layering of heat from ceiling to floor - can predict flashover (Huge) based on the changing of the heat layers. It’s generally pretty solid lines - 200, 500, 800: no one survives above 212F; even with PPE. 800 is when it’s hot enough that a flashover can happen with oxygen and fuel). (Thermal layers)

[0060] With edge detection, if properly configured, can detect hole in floor / ceiling (e.g. line that ‘shouldn’t be there’ may be indication of a hole).

[0061] Horizontal lines might help give the idea of a staircase (successive horizontal lines); rectangle (vert.) = doorway.

[0062] In one example, using an on-board accelerometer and thermal imaging systems described herein, systems described herein may recognize that the user is looking at themselves in a reflective surface, such as a mirror.

[0063] Object recognition algorithms described herein may be able to identify a number of items of interest for a user. For example, obstacles 322 may include boxes, barrels or other items that may block, or partially block a pathway 324. Obstacles 322 may also include hazardous items, such as flammable or explosive items. Object recognition algorithms may also identify a pathway 324. Pathway 324 may include a doorway with or without a door, a window, a hallway, etc. Object recognition algorithms described herein may be able to identify victims 326, which may include human or animals. Other objects 328 may also be identified. For example, a coffee table in the middle of a room may be identified so that a firefighter does not run into, or trip over, it.

[0064] As noted above, in-mask display systems may be capable of displaying a lot of information simultaneously - identified object indications and / or identified edges on a thermal or regular video feed of a scene. Additionally, indicia about a user’s equipment - SCBA battery life, remaining air supply, etc. may be displayed either on the same screen as the video feed or elsewhere in a heads up display configuration. Further, sensor signal information related to the user’s position, as well as potential objectives (e.g. a man down, a victim, a hazardous item, etc.) may also be displayed.

[0065] Systems and methods herein, as indicated in block 330, include generating a user interface for presentation on a display. The generated user interface may be different for different users - e.g. a supervisor may see a user interface with more information than a user interface generated for a fire fighter or other first responder, as the supervisor may be under less stress and able to process more visual information. In a first responder view, only a subset of the identified edges and / or objects may be indicated.

[0066] For example, items indicating hazards or dangers may be prioritized, or a detected victim may be prioritized, or an obstacle in the pathway may be prioritized. An alert 332 may be generated that indicates the prioritized item. The alert 332 may include an alphanumeric indication - e.g. “NO GO” or “Ammunition Detected” or “Man Down” - or another indication. For example, the detected object 336 may be indicated - for example it may be outlined, or highlighted or otherwise emphasized. The image feed 334 may also be presented. Scene analysis such as alerts or detected objects may be presented as a composite image feed 334, or may be presented separately, e.g. in a different display, a different portion of a display, or in another suitable manner so as to draw attention to the subset of items detected. The user interface may also include other information or indicia 338.

[0067] In another example, a view presented to a user is different than a view presented to a supervisor. The supervisor view may include everything, with an ability to select something of interest. For a supervisor, having the full view may be particularly useful if the user is lost. For a firefighter, systems herein may highlight only items that may be of immediate concern such as drums of flammable liquid, windows / doors, other people. Systems herein may also highlight unidentifiable things that look like they may be hazardous, for example bags of chemicals; something smoldering or a pipe leaking.

[0068] In some embodiments, the view presented to the user includes an overlay applied to the image. For example, a supervisor may view a live stream of captured thermal images and identify an object with a label that is unusually warm. The label may not be clear to, or noticed by the firefighter. The supervisor may be able to identify the label - for example by asking an evacuee, from a building plan, or in another matter. The supervisor may communicate back to the firefighter by generating an overlay with the label displayed more clearly, more prominently, or otherwise called out in an overlay over the thermal image.

[0069] It is expressly contemplated that object identification may come from sources other than edge detection and comparison to an object database. Other information may be captured from other sources. For example, a location of where smoke is exiting a building may help to identify an item that a fire inside the building sees.

[0070] Further, because systems herein can identify specific items as, for example, a door or a barrel, systems herein may also be responsive to an audible command from the user. For example, a user may say “show me all doorways and windows” and systems herein may highlight all identified doorways and windows. The user may also be trying to identify, for example “what is making that light?” and systems herein may respond.

[0071] While a user interface is described with respect to a display, it is expressly contemplated that, based on the identified objects, alerts or other information may be communicated to a user through another medium. For example, haptic feedback could be used to indicate a direction the user should go or a deviation from a pathway. Audible feedback, such as alerts, instructions or directions, may be provided using a speaker.

[0072] The user interface may be communicated, as indicated in block 340, to a device with a display. In some embodiments, image analysis of athermal or other image feed is conducted using a processing device associated with a user - e.g. a processor within an SCBA, PAPR, etc. In some embodiments, image analysis is done by the image capturing device - e.g. thermal imager 304 or other imaging device 306, and then provided to the in-mask display. In some embodiments, image data is sent from thermal imager and / or other imaging device to a remote processor, using a wireless or cloud-based network, and the remote processor conducts the scene analysis and communicates the user interface. The user interface may be communicated back to a device associated with the thermal and / or other imager - e.g. an in-mask display coupled to, or associated with, the imaging system. The user interface may also be communicated to a separate device - e.g. a display associated with a site manager or supervisor.

[0073] Any one of the image capture process and / or edge detection described herein may be part of another imaging system such as a wearable either in the mask, on the user, or in an SCBA electronics package. The imaging system may be configured to display a composite image (i.e., including the captured image and information associated with the edge detection). For example, a user may see a live image with the identified objects or alerts overlaid in a display component associated with the mask. The images and / or identified objects may be made available to remote viewers (e.g., a commander) via any suitable communication interface, e.g., by wireless transmission. Thermal images may be selectively captured in a local, remote or cloud-based storage, e.g., based on a quality detection process, triggered by the user and / or command, and may be used for learning in a separate system.

[0074] In another example, mask assembly 210 may be worn by a firefighter that communicates, such as, with another firefighter, also wearing a mask assembly. Mask assembly 210 may be configured to transmit an image (such as a composite image including one or more layers and / or a thermal image and / or edge detection information associated with the thermal image) to the corresponding display of the second fire fighter, which may display the generated and transferred composite image transmitted.

[0075] In some embodiments, the received image is a portion of a video feed. It is expressly contemplated that method 300 can be applied in real-time such that an augmented, or composite image feed is presented to a user without a noticeable delay.

[0076] FIGS. 4-9 illustrate example user interfaces that may be presented to a user in accordance with embodiments herein.

[0077] FIG. 4 illustrates an example composite thermal image feed 400 that may be presented on an in-mask display, a heads-up display, or otherwise presented to a user. A door 410 is detected, with a doorknob 420. Based on the thermal image feed, it is determined that the area behind the door is too hot to enter, therefore an alert 430 is presented. Other sensor information 440 may also be presented, such as an ambient temperature, battery life, air supply, etc.

[0078] However, while FIG. 4 illustrates a scenario where visibility is good, it is expressly contemplated that systems and methods described herein can be used in low-visibility scenarios to provide more information to a first responder, such as a firefighter.

[0079] FIG. 5 illustrates a thermal image 500 captured by a user wearing a thermal imaging device. FIG. 5 illustrates the difficulty that users can have when trying to interpret thermal images. Two individuals 510 can be seen, but not readily identified based on the image. A structure 520 is visible in the background, but also not readily identifiable. There are heat artifacts 530 that make it difficult to determine additional details of structure 520 and, potentially more importantly, obstacles that may be present on the ground between a user and structure 520.

[0080] FIGS. 6A-6B illustrate user interfaces that may be presented to a user of an in-mask display system in accordance with embodiments herein. FIG. 6A illustrates a user interface 600 that includes an image feed captured by a thermal imaging device. Another individual 610 can be clearly seen. However, the rest of the scene is difficult to identify. An unknown item 620 is illustrated. Unknown item 620 may represent a heat artifact or an actual obj ect that presents a hazard to individual 610 or to a user viewing user interface 600 using an in-mask display system. Systems and methods herein, using machine learning algorithms trained on a set of training data may be able to identify unknown item 620.

[0081] FIG. 6B illustrates a composite image feed where unknown item has been identified as another firefighter. A respirator 622 and / or a helmet 624 may be detectable and, based on said detection, firefighter 620 may be identified.

[0082] FIG. 7 illustrates an example user interface for an in-mask display component. As illustrated, user interface 700 includes a lot of information - potentially more information than a user in a high stress situation, such as a rescue operation in a building on fire, can process. An identity 702 of another individual may be retrieved, for example, using a suitable wireless protocol, and displayed. An evacuation icon 706 may be presented based on a floor plan constructed using fiducials, retrieved from building plans, or using another suitable method. A path to the exit 704 may also be provided. Detected edges 710, 712 may also be presented. The user is also being alerted 720 that an air supply is below 50%.

[0083] Systems and methods herein, in some embodiments, selectively provide a subset of detected objects and potential alerts so as not to overwhelm a user. For example, either alert 706 or path 704 may be presented. Edge 710 may be presented, while edge 712 is left off.

[0084] What items are selectively prioritized may be customizable. For example, firefighters in an area with frequent forest fighters may see different alerts than firefighters that generally respond to fires in a downtown area. Both may see different images than emergency responders in an industrial area. In some embodiments, systems and methods herein may have algorithms trained on different training data, for example, based on anticipated use. In some embodiments herein, systems and methods herein may have different selection algorithms that prioritize different alerts. Systems and methods herein may be able to, in some embodiments, identify a scene as a residential home and selectively prioritize different alerts than if the scene was identified as a forest.

[0085] FIG. 8A illustrates another example of the difficulty users face in determining their surroundings in a low-visibility environment. A user, as illustrated in FIG. 8A, has received an alert 802 that another first responder is down, e.g. in distress or otherwise in need of aid. The signal strength may be useful for determining whether or not the user is getting closer to, or farther away from, the downed responder. However, while work has been done in the area of locating and directing a user to a downed comrade, directions are generally limited to what information is available. A distance from the downed individual is calculatable based on the signal strength, but that does not account for walls and / or obstacles. For example, a path may be generated using fiducials or by referencing a floor plan. Systems and methods herein improve information that can be provided to an individual by detecting debris in between a user and the downed man, and providing appropriate alerts or composite images.

[0086] FIG. 8B illustrates a number of additional indications that can be provided using systems and methods herein. For example, two different obstacles 870 are detected and identified. The identity labels 872 are presented. Additional information may also be provided - for example barrels may indicate a hazardous substance. Systems and methods herein may detect a “flammable” symbol on one or more of the barrels and provide an alert. Alternatively, systems and methods herein may detect signs indicative of explosive material present in the barrels - e.g. posted signs regarding regulatory requirements such as OSHA compliance in the US, for example.

[0087] A fire source indication 860 may be provided. For example, the illustrated scene may be full of smoke. The fire source indication 860 may indicate an area on fire, based on thermal imagery, or may indicate a source of the fire, for example based on a heat level detected from the thermal imagery.

[0088] A path 862 may be outlined. The illustrated scene shows a bridge with railings. Path 862 may be indicated to ensure that the user does not stray too close to a railing 864 and accidentally fall over.

[0089] FIG. 9 illustrates a schematic of a user interface that may be presented to a user in accordance with embodiments herein. FIG. 9 is a schematic view of a user interface 900 that may be presented to a user of an SCBA unit, for example. A distress signal may be detected, and a strength 902 of the distress signal may be presented. Strength bars are illustrated in FIG. 9, however other illustrative indications may be presented, such as the percentage signal illustrated in FIG. 8. In some examples, the strength bars may be color coded in a range of different colors indicative of the strength of the distress signal 902. For example, the strength bars may be highlighted with green (lowest strength), followed by yellow (intermediate strength), and then red (highest strength). Pathway options 904 may be illustrated, as well as walls 906. Detected obstacles, such as table 908 may also be indicated.

[0090] FIG. 10 illustrates a scene analyzing system in accordance with embodiments herein. Some components of system 1000 are illustrated as remote from others. However, it is expressly contemplated that this is for illustrative purposes only. The illustrated components may be distributed differently in some embodiments, or as part of a single device, in some embodiments.

[0091] A Self-Contained Breathing Apparatus (SCBA) 1010 includes a display unit 1008 that has a user interface. The user interface may include a graphical user interface component. In some embodiments the user interface includes indicators - such as indicator lights or icons indicative of, for example, equipment performance, user location, signal strength relative to other users, alerts, etc. In some embodiments, the SCBA also includes a microphone 1002 and a speaker 1004 for communication between individuals. SCBA 1010 may also include other components 1010, e.g. those described in FIGS. 2A-2B, air circulation system, filtration systems, etc. Additionally, while SCBA 1000 is illustrated, it is expressly contemplated that systems described herein may be implemented in other personal protective equipment, such as PAPRs, for example.

[0092] System 1000 also includes one or more thermal imaging systems 1020. Each thermal imaging system includes an image capture device 1022 configured to capture images in the infrared spectrum. Specifically, image capturing device is configured to capture images using infrared radiation emitted by an object. In some embodiments, image capturing device detects radiation in the long-infrared wavelength range, e.g. between 9-14 pm.

[0093] Thermal imaging system 1020 also includes a mount 1026 such that it can attach to an individual, their clothing or their equipment. Thermal imaging system 1020 may also include other components 1028. In embodiments where thermal imaging system 1020 is part of a device separate from either or both of SCBA 1010 and image processing system 1040, system 1020 also includes an image communication component 1024, which may transmit image data through a wired or wireless network protocol.

[0094] In some embodiments, thermal imaging system 1020 communicates with SCBA 1010 through scene analyzing system, e.g. all image data is processed prior to being displayed on a graphical user interface of display unit 1008. However, it is expressly contemplated that at least some image data may be transmitted directly for display on display unit 1008.

[0095] System 1000 also includes an image processing system 1040. Image processing system 1040 is illustrated as separate from SCBA 1010 and thermal imaging system 1020, however it is expressly contemplated that, in some embodiments, at least some functionality of image processing system 1040 may be completed by a processor associated with either SCBA 1010 or thermal imaging system 1020.

[0096] Image processing system 1040 includes a data receiver 1041. Data receiver 1041 may receive image data from thermal imaging systems and sensor data from sensors 1030. In some embodiments, sensors 1030 include environmental sensors such as fiducials placed in an environment, temperature sensors, etc. Sensors 1030 may also include equipment sensors, in some embodiments, e.g. battery life sensor, air pressure sensor, signal strength sensor, etc. Sensors 1030 may also include individual- related sensors, such as breath or heart rate monitors.

[0097] Image processing system 1040 processes the data received from data receiver 1041 in realtime, or substantially real-time such that a user does not notice a delay, using one or more processing algorithms. Datastore 1070 is illustrated as separate from image processing system 1040. However, it is expressly contemplated that datastore 1070 may be stored in a memory associated with the image processing system 1040. Either or both of datastore 1070 and image processing system 1040 may be incorporated into a physical device or accessible over a wireless or cloud network protocol. Image processing may utilize one or more processing algorithms stored in datastore 1070. While not shown, it is expressly contemplated that processing received image data may include the use of enhancement, restoration, encoding and / or compression algorithms.

[0098] Object detector 1044 may identify one or more objects in an image. For example, edge detection algorithms 1074 may be used to detect edge or edge indicia in an image. Object detector 1044 may then, using an object identification algorithm 1076, determine whether the detected edges are an object, and what the object is - e.g. a table, a chair, a box or a barrel, etc. Object identification algorithm 1076 may be a trained machine-learning algorithm. Algorithm 76 may be trained based on a training dataset that includes images of objects. Training data may include images of objects in different lighting conditions, in some embodiments. Training data may include images of objects obscured, for example by smoke or haze. Object detector 1044 may identify the object by comparing image data to an object datastore 1072 using algorithm 1076.

[0099] In some embodiments, object detector 1044 uses output from an edge detection algorithm 1074, which may detect edges of potential objects that object detector 1044 can then identify as an object. However, it is expressly contemplated that, in some embodiments, object identifier algorithm identifies one or more objects from image data without edge detection data.

[0100] Object detector 1044 may identify objects by referencing an object datastore 1072. Object datastore 1072 may also receive data for future use based on the object detector successfully or unsuccessfully identifying an object. Object datastore 1072 may include images of objects in a variety of different environments, in some embodiments.

[0101] Object detector 1044 may receive temperature information about an object from thermal gradient detector 1042. For example, a mannequin does not emit as much heat radiation as a human being. Thermal gradient information may be very helpful in identifying victims - such as humans or pets - especially if said victim is unconscious or hidden.

[0102] Once one or more objects are detected, object selector 1048 determines which objects should be highlighted or otherwise called out in a composite image. One or more selection algorithms 1082 may be applied to determine whether an object should or should not have attention called to it. Selection algorithms may be non-machine learning models, for example basing selection of an object off of preset priorities - e.g. using rules such as humans are always called out, debris outside of a path are not, signs indicating flammable material are always called out and paired with an alert, etc.

[0103] Object selector 1048 may select objects in response to an external stimuli. For example, a supervisor may be streaming video in real-time (or substantially real-time), and may indicate, either audibly, haptically (e.g. by touching a touchscreen) or in another suitable manner, an item that should be identified. A supervisor may see unidentifiable debris, for example, and may indicate a desire to know more about what potential object may be present.

[0104] However, it is expressly contemplated that an object selection algorithm 1082 is a trained machine-learning algorithm in some embodiments. Object selector 1048 may also select how a particular object should be called out to a user. For example, the object may be outlined or shaded, alphanumeric text or symbols may be generated for the center of the screen or for placement over or near the object, and / or an audible or haptic feedback or alert signal may be generated. These and other suitable call-out indications may be included. Object selector 1048 may generate a callout in different colors - for example flammable material may always be indicated with red while victims may always be indicated in green.

[0105] For example, it may be desired for systems herein to prioritize humans or human-related objects such as a hand or a leg. Someone may fall through and be stuck between floors and systems herein may identify and alert the user to a detected top half or bottom half of an individual. Systems and methods herein may be able to identify someone kneeling, crawling, holding a hose, wearing or not wearing SCBA equipment, partially submerged or covered by standing water or debris. Training data may include a number of possible alignments of humans and / or body parts.

[0106] Image processing system 1040 may generate a composite image, using composite image generator 1054, that includes the object callout indication and the thermal image data. In some embodiments, for example when site information 1078 is available, a map generator 1052 may generate path indications for inclusion in the composite image.

[0107] Graphical user interface generator 1056 generates a graphical user interface based on the generated composite image. The generated graphical user interface may also include indications received from sensor(s) 1030 in some embodiments. A communication component 1058 communicates the generated graphical user interface to display unit 1008. In some embodiments, the graphical user interface generator 1056 generates a first graphical user interface for display on display unit 1008 and a second, different, graphical user interface for display on a second device 1060.

[0108] The term “machine-learning,” as described herein, is used broadly to refer to any number of algorithm that is capable of learning from data without pre-defined rules or models. Machine learning models described herein may include regression models, classification models, clustering models, dimensionality reduction models, ensemble method models, neural net or deep learning models, transfer learning models, reinforcement learning models, natural language processing models or word embedding models. As used herein, machine learning may refer to either a supervised or unsupervised model. Data receiver 1041 may also retrieve information from datastore 1070 that may be relevant for a particular operation. For example, site information 1078 may be available. For example, building plans of a home or school, map of a worksite, or other information that may indicate pathways. Data receiver 1041 may also receive other information 1080. Other information may include information communicated from another device 1060. For example, a sensor 1030 may detect a man-down, and transmit a signal that can be used to locate the downed man. Other device 1060 may be another SCBA 1010, a device associated with a supervisor, site manager, a second first responder, etc.

[0109] FIG. 11 is a block diagram providing an operating perspective of an image processing system 1106 when hosted as a cloud-based platform. As described herein, image processing system 1106, in some embodiments, is capable of supporting multiple, distinct access points 1115 and / or work environments having an overall population of workers, as described with respect to FIG. 1, for example. In the example of FIG. 11, the components of the image processing system are arranged according to multiple logical layers that implement the techniques of the disclosure. Each layer may be implemented by one or more modules and may include hardware, software, or a combination of hardware and software.

[0110] In some example approaches, computing devices 1132, display 1112, input devices 1134, and / or safety stations 1115 operate as clients 1130 that communicate with PPE fit testing system 1106 via interface layer 1136. Computing devices 1132 typically execute client software applications, such as desktop applications, mobile applications, and / or web applications. Computing devices 1132 may represent any of the computing devices described herein, either built into the SCBA or freestanding. Examples of computing devices 1132 may include, but are not limited to, a portable or mobile computing device (e.g., smartphone, wearable computing device, tablet), laptop computers, desktop computers, smart television platforms, and / or servers.

[0111] In some example approaches, computing devices 1132, display 1112, sensors (e.g., cameras, scanners, barcode or other) 1122, input devices 1134 and / or image capture devices 1115 may communicate with image processing system 1106 to send and receive information related to PPE fit testing, PPE fit verification, alert generation, or the like. Client applications executing on computing devices 1132 may communicate with system 1106 to send and receive information that is retrieved, stored, generated, and / or otherwise processed by functionalities 1140. In some examples, client applications may request and display information received or generated by system 1106, such as captured thermal images with or without overlays or annotations. In addition, the client applications may interact with system 1106 to query for analytics information about objects detected and / or identified. The client applications may output for display information received from system 1106 to visualize such information for users of clients 1130. Additionally, as described herein, system 1106 may provide an interactive experience for users using client applications.

[0112] Client applications executing on computing devices 1132 may be implemented for different platforms but include similar or the same functionality. For instance, a client application may be a desktop application compiled to run on a desktop operating system, such as MicrosoftWindows, Apple OS X, or Linux, to name only a few examples. As another example, a client application may be a mobile application compiled to run on a mobile operating system, such as Google Android, Apple iOS, Microsoft Windows Mobile, or BlackBerry OS to name only a few examples. As another example, a client application may be a web application such as a web browser that displays web pages received from system 1106. In the example of a web application, system 1106 may receive requests from the web application (e.g., the web browser), process the requests, and send one or more responses back to the web application. In this way, the collection of web pages, the client-side processing web application, and the server-side processing performed by 1106 collectively provides the functionality described herein.

[0113] In some examples, interface layer 1136 may provide Representational State Transfer (RESTful) interfaces that use HTTP methods to interact with services and manipulate resources of systems 1106. In such examples, functionalities 1140 may generate JavaScript Object Notation (JSON) messages that interface layer 1136 sends back to the client application that submitted the initial request. In some examples, interface layer 1136 provides web services using Simple Object Access Protocol (SOAP) to process requests from client applications. In still other examples, interface layer 1136 may use Remote Procedure Calls (RPC) to process requests from clients 1130. Upon receiving a request from a client application to use one or more functionalities 1140, interface layer 736 sends the information to application layer 1138, which includes functionality 1140.

[0114] As shown in FIG. 11, system 1106 also includes an application layer 1138 that represents a collection of services for implementing much of the underlying operations of system 1106. Application layer 1138 receives information included in requests received from client applications that are forwarded by interface layer 1136 and processes the information received according to one or more of services 1140 invoked by the requests. Application layer 1138 may be implemented as one or more discrete software services executing on one or more application servers, e.g., physical or virtual machines. That is, the application servers provide runtime environments for execution of services 1140. In some examples, the functionality of interface layer 1136 as described above and the functionality of application layer 1138 may be implemented at the same server.

[0115] Application layer 1138 may include one or more separate software functionalities 1140 (e.g., processes) that may communicate via, for example, a logical service bus 1144. Service bus 1144 generally represents a logical interconnection or set of interfaces that allows different services to send messages to other services, such as by a publish / subscription communication model. For example, each of services 1140 may subscribe to specific types of messages based on criteria set for the respective service. When a service publishes a message of a particular type on service bus 1144, other services that subscribe to messages of that type will receive the message. In this way, each of services 1140 may communicate information to one another. As another example, services 1140 may communicate in point-to-point fashion using sockets or other communication mechanism. Before describing the functionality of each of services 1140, the layers are briefly described herein.

[0116] Data layer 1146 of system 1106 includes one or more datastores 1148. A datastore, generally, may be any data structure or software that stores and / or manages data; a datastore may store data in structured or unstructured form. For example, a datastore may be a remote datastore hosted by one or more remote compute servers coupled to the controller by a packet-switched network. As another example, the datastore may be locally cached within a computer-readable medium of the computing device. Example datastores include but are not limited to one or more of database management systems, online analytical processing database, map, table, hash tables, or any other suitable structure for storing data. In one example approach, data layer 1146 of system 1106 includes datastores 1148 used to provide persistence for information in system 1106. Data layer 1146 may be implemented, for instance, using Relational Database Management System (RDBMS) software to manage information in datastores 1148. The RDBMS software may manage one or more datastores 1148, which may be accessed using Structured Query Language (SQL). Information in the one or more databases may be stored, retrieved, and modified using the RDBMS software. In some examples, data layer 1146 may be implemented using an Object Database Management System (ODBMS), Online Analytical Processing (OLAP) database, or any other suitable data management system.

[0117] In the example of FIG. 11, each of services 1140A-11401 is implemented in a modular form within system 1106. Although shown as separate modules for each service, in some examples the functionality of two or more services may be combined into a single module or component. Each of functionalities 1140 may be implemented in software, hardware, or a combination of hardware and software. Moreover, functionalities 1140 may be implemented as standalone devices, separate virtual machines or containers, processes, threads, or software instructions generally for execution on one or more computer processors or processing circuitry.

[0118] Although generally described herein as captured images, identified objects, object images and / or training data, or any other stored information described herein being stored in datastores 1148, in some examples, datastores 1148 may additionally or alternatively include data representing said data or any other stored information described herein. As one example, encoded lists, vectors, or the like representing a previously stored image may be stored in addition to, or as an alternative, the previously stored image itself. In some examples, such data representing user data, captured images, sensed sensor signals, or any other stored information described herein may be simpler to store, evaluate, organize, categorize, or the like in comparison to storage of the actual user data, image data, or the like.

[0119] FIG. 12 is a block diagram of an image processing system architecture. The remote server architecture 1200 illustrates one embodiment of an implementation of a PPE fit test system 1210. As an example, remote server architecture 1200 can provide computation, software, data access, and storage services that do not require end-user knowledge of the physical location or configuration of the system that delivers the services. In various embodiments, remote servers can deliver the services over a wide area network, such as the internet, using appropriate protocols. For instance, remote servers can deliver applications over a wide area network and they can be accessed through a web browser or any other computing component. Software or components shown or described in FIGS. 1-11 as well as the corresponding data, can be stored on servers at a remote location. The computing resources in a remote server environment can be consolidated at a remote data center location or they can be dispersed. Remote server infrastructures can deliver services through shared data centers, even though they appear as a single point of access for the user. Thus, the components and functions described herein can be provided from a remote server at a remote location using a remote server architecture. Alternatively, they can be provided by a conventional server, installed on client devices directly, or in other ways.

[0120] In the example shown in FIG. 12, some items are similar to those shown in earlier figures. FIG. 12 specifically shows that system 1210 can be located at a remote server location 1202. Therefore, computing device 1220 accesses those systems through remote server location 1202. Operator 1250 can use computing device 1220 to access user interfaces 1222 as well. While a single operator 1250 is illustrated as connecting to system 1210, it is expressly contemplated that multiple operators may access system 1210, using either device 1220 or another suitable device.

[0121] FIG. 12 depicts a remote server architecture. FIG. 12 shows that it is contemplated that some elements of systems described herein are disposed at remote server location 1202 while others are not. By way of example, storage 1230, 1240 or 1260 or module selector 1270 can be disposed at a location separate from location 1202 and accessed through the remote server at location 1202. Regardless of where they are located, they can be accessed directly by computing device 1220, through a network (either a wide area network or a local area network), hosted at a remote site by a service, provided as a service, or accessed by a connection service that resides in a remote location. Also, the data can be stored in substantially any location and intermittently accessed by, or forwarded to, interested parties. For instance, physical carriers can be used instead of, or in addition to, electromagnetic wave carriers.

[0122] It will also be noted that the elements of systems described herein, or portions of them, can be disposed on a wide variety of different devices. Some of those devices include servers, desktop computers, laptop computers, imbedded computer, industrial controllers, tablet computers, or other mobile devices, such as palm top computers, cell phones, smart phones, multimedia players, personal digital assistants, etc.

[0123] FIGS. 13-14 show examples of mobile devices that can be used in the embodiments shown in previous Figures.

[0124] FIG. 13 is a simplified block diagram of one illustrative example of a handheld or mobile computing device that can be used as a user's or client's handheld device 1316 (e.g., as computing device 1320 in FIG. 13), in which the present system (or parts of it) can be deployed. For instance, a mobile device can be deployed in the operator compartment of computing device 1320 for use in generating, processing, or displaying the data. FIG. 14 is another example of a handheld or mobile device.

[0125] FIG. 13 provides a general block diagram of the components of a client device 1316 that can run some components shown and described herein. Client device 1316 interacts with them, or runs some and interacts with some. In the device 1316, a communications link 1313 is provided that allows the handheld device to communicate with other computing devices and under some embodiments provides a channel for receiving information automatically, such as by scanning. Examples of communications link 1313 include allowing communication though one or more communication protocols, such as wireless services used to provide cellular access to a network, as well as protocols that provide local wireless connections to networks.

[0126] In other examples, applications can be received on a removable Secure Digital (SD) card that is connected to an interface 1315. Interface 1315 and communication links 1313 communicate with a processor 1317 (which can also embody a processor) along a bus 1319 that is also connected to memory 1321 and input / output (I / O) components 1323, as well as clock 1325 and location system 1327.

[0127] I / O components 1323, in one embodiment, are provided to facilitate input and output operations and the device 1316 can include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors and output components such as a display device, a speaker, and or a printer port. Other I / O components 1323 can be used as well. Clock 1325 illustratively comprises a real time clock component that outputs a time and date. It can also provide timing functions for processor 1317.

[0128] Illustratively, location system 1327 includes a component that outputs a current geographical location of device 1316. This can include, for instance, a global positioning system (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning system. It can also include, for example, mapping software or navigation software that generates desired maps, navigation routes and other geographic functions.

[0129] Memory 1321 stores operating system 1329, network settings 1331, applications 1333, application configuration settings 1335, data store 1337, communication drivers 1339, and communication configuration settings 1341. Memory 1321 can include all types of tangible volatile and non-volatile computer-readable memory devices. It can also include computer storage media (described below). Memory 1321 stores computer readable instructions that, when executed by processor 1317, cause the processor to perform computer-implemented steps or functions according to the instructions. Processor 1317 can be activated by other components to facilitate their functionality as well.

[0130] FIG. 14 shows that the device can be a smart phone 1471. Smart phone 1471 has a touch sensitive display 1473 that displays icons or tiles or other user input mechanisms 1475. Mechanisms 1475 can be used by a user to run applications, make calls, perform data transfer operations, etc. In general, smart phone 1471 is built on a mobile operating system and offers more advanced computing capability and connectivity than a feature phone.

[0131] Note that other forms of the devices 1471 are possible.

[0132] FIG. 15 is a block diagram of a computing environment that can be used in embodiments shown in previous Figures.

[0133] FIG. 15 is one example of a computing environment in which elements of systems and methods described herein, or parts of them (for example), can be deployed. With reference to FIG. 15, an example system for implementing some embodiments includes a general-purpose computing device in the form of a computer 1510. Components of computer 1510 may include, but are not limited to, a processing unit 1520 (which can comprise a processor), a system memory 1530, and a system bus 1521 that couples various system components including the system memory to the processing unit 1520. The system bus 1521 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. Memory and programs described with respect to systems and methods described herein can be deployed in corresponding portions of FIG. 15. Computer 1510 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 1510 and includes both volatile / nonvolatile media and removable / non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media is different from, and does not include, a modulated data signal or carrier wave. It includes hardware storage media including both volatile / nonvolatile and removable / non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 1510. Communication media may embody computer readable instructions, data structures, program modules or other data in a transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0134] The system memory 1530 includes computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 1531 and random access memory (RAM) 1532. A basic input / output system 1533 (BIOS) containing the basic routines that help to transfer information between elements within computer 1510, such as during start-up, is typically stored in ROM 1531. RAM 1532 typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by processing unit 1520. By way of example, and not limitation, FIG. 15 illustrates operating system 1534, application programs 1535, other program modules 1536, and program data 1537.

[0135] The computer 1510 may also include other removable / non-removable and volatile / nonvolatile computer storage media. By way of example only, FIG. 15 illustrates a hard disk drive 1541 that reads from or writes to non-removable, nonvolatile magnetic media, nonvolatile magnetic disk 1552, an optical disk drive 1555, and nonvolatile optical disk 1556. The hard disk drive 1541 is typically connected to the system bus 1521 through a non-removable memory interface such as interface 1540, and optical disk drive 1555 are typically connected to the system bus 1521 by a removable memory interface, such as interface 1550.

[0136] Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (e.g., ASICs), Application-specific Standard Products (e.g., ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0137] The drives and their associated computer storage media discussed above and illustrated in FIG. 15, provide storage of computer readable instructions, data structures, program modules and other data for the computer 1510. In FIG. 15, for example, hard disk drive 1541 is illustrated as storing operating system 1544, application programs 1545, other program modules 1546, and program data 1547. Note that these components can either be the same as or different from operating system 1534, application programs 1535, other program modules 1536, and program data 1537.

[0138] A user may enter commands and information into the computer 1510 through input devices such as a keyboard 1562, a microphone 1563, and a pointing device 1561, such as a mouse, trackball or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite receiver, scanner, or the like. These and other input devices are often connected to the processing unit 1520 through a user input interface 1560 that is coupled to the system bus, but may be connected by other interface and bus structures. A visual display 1591 or other type of display device is also connected to the system bus 1521 via an interface, such as a video interface 1590. In addition to the monitor, computers may also include other peripheral output devices such as speakers 1597 and printer 1596, which may be connected through an output peripheral interface 1595.

[0139] The computer 1510 is operated in a networked environment using logical connections, such as a Local Area Network (LAN) or Wide Area Network (WAN) to one or more remote computers, such as a remote computer 1580.

[0140] When used in a LAN networking environment, the computer 1510 is connected to the LAN 1571 through a network interface or adapter 1570. When used in a WAN networking environment, the computer 1510 typically includes a modem 1572 or other means for establishing communications over the WAN 1573, such as the Internet. In a networked environment, program modules may be stored in a remote memory storage device. FIG. 15 illustrates, for example, that remote application programs 1585 can reside on remote computer 1580.

[0141] It will be apparent to those skilled in the art that the specific exemplary embodiments, elements, structures, features, details, arrangements, configurations, etc., that are disclosed herein can be modified and / or combined in numerous ways. In summary, numerous variations and combinations are contemplated as being within the bounds of the conceived invention, not merely those representative designs that were chosen to serve as exemplary illustrations. Thus, the scope of the present invention should not be limited to the specific illustrative structures described herein, but rather extends at least to the structures described by the language of the claims, and the equivalents of those structures. Any of the elements that are positively recited in this specification as alternatives may be explicitly included in the claims or excluded from the claims, in any combination as desired. Any of the elements or combinations of elements that are recited in this specification in open-ended language (e.g., comprise and derivatives thereof), are considered to additionally be recited in closed-ended language (e.g., consist and derivatives thereof) and in partially closed-ended language (e.g., consist essentially, and derivatives thereof). Although various theories and possible mechanisms may have been discussed herein, in no event should such discussions serve to limit the claimable subject matter. To the extent that there is any conflict or discrepancy between this specification as written and the disclosure in any document that is incorporated by reference herein but to which no priority is claimed, this specification as written will control.

[0142] A thermal imaging system configured for use with a respirator mask is presented that include an in-mask display unit configured to be mounted within an interior volume of a respirator mask and including an in-mask micro-display. The system also includes a wearable thermal imaging camera that is configured to capture a thermal image and to communicate at least a portion of the captured thermal image to the in-mask micro-display. The system also includes an image processing system including a processor and memory, the system being configured to: detect an object in the thermal image, generate a callout to indicate the detected object, and compile a composite image, the composite image includes the thermal image and the generated callout.

[0143] The system may be implemented such that the image processing system is configured to detect the object using an object detector configured to, using a database of objects, identify the detected object.

[0144] The system may be implemented such that the image processing system is configured to detect edges in the thermal image, using an edge detection algorithm, and the object detector detects the image based on the detected edges identifies the detected object by comparing the detected edges to the database of objects.

[0145] The system may be implemented such that the object detector includes a trained machinelearning algorithm.

[0146] The system may be implemented such that the detected object is a first detected object, and the image processing system is further configured to: detect a second object in the thermal image, select the first object, and the generated callout indicates the first object. The second object is not called out in the composite image. The system may be implemented such that the callout includes: a highlight or a shading of the detected object, an outline the detected object, a label identifying the detected object, an overlay applied over the image, or an alert.

[0147] The system may be implemented such that the object detector is configured to identify the detected object based on a detected thermal gradient between the object and its surroundings.

[0148] The system may be implemented such that the image processing system includes a data receiver that receives a sensor signal from a sensor other than the thermal imaging system.

[0149] The system may be implemented such that the display unit is a micro-display and includes a control system for controlling the in-mask micro-display and the wearable thermal imaging camera. The wearable thermal imaging camera includes a wide field of view so that the captured thermal image includes at least a primary subimage and a secondary subimage. The control system is configured so that the in-mask micro-display presents a default view in which the in-mask micro-display shows the primary subimage of the captured thermal image but does not show the secondary subimage of the captured thermal image. Upon the control system receiving an input to shift the in-mask micro-display from the default view to an auxiliary view, the control system causes at least a portion of the primary subimage to be directionally displaced on the in-mask micro-display by the secondary subimage so that the secondary subimage is now shown in place of the displaced portion of the primary subimage.

[0150] The thermal imaging system may be implemented such that the in-mask display unit includes a monocular eyepiece inside which the in-mask micro-display is mounted, and, the in-mask display unit further includes an objective lens and an ocular lens that are mounted in the monocular eyepiece rearward of the in-mask micro-display and that combine with the in-mask micro-display to form an optical pathway to deliver images to a single eye of the user.

[0151] A respirator mask including the thermal imaging system described herein, where the respirator mask defines an interior volume when fitted to the face of a human user.

[0152] The respirator mask may be implemented such that the wearable thermal imaging camera is configured to be attached to the outside of the respirator mask and is wirelessly connected to the control system.

[0153] The respirator mask may be implemented such that the wearable thermal imaging camera is configured to be attached to a protective helmet or a protective garment that is worn in conjunction with the respirator mask, and the wearable thermal imaging camera is wirelessly connected to the control system.

[0154] A method of providing thermal images to a wearer of a respirator mask is presented that includes capturing a thermal image using a thermal imaging system, processing the thermal image, using an image processing system. Processing includes detecting an object in the thermal image, generating an object callout for the detected object, generating a composite image including the thermal image and the detected object, and generating a graphical user interface for a display component of the respirator mask, the graphical user interface including the composite image, and displaying the graphical user interface on a display coupled to the respirator mask.

[0155] The method may be implemented such that the thermal imaging system includes a thermal imaging camera mounted to a self-contained breathing apparatus that includes the respirator mask.

[0156] The method may be implemented such that detecting includes an object detector comparing the thermal image to a datastore of reference objects.

[0157] The method may be implemented such that processing the thermal image further includes detecting an edge, using an edge detection algorithm, and the object detector detects the image based in part on the detected edge.

[0158] The method may be implemented such that the object detector includes a trained machine learning algorithm.

[0159] The method may be implemented such that the method further includes, based on the comparison, identifying the object.

[0160] The method may further include detecting a thermal gradient between the detected object and its surroundings in the thermal image, and the object is identified based on the thermal gradient.

[0161] The method may be implemented such that the detected object is a first detected object, and the method further includes: detecting a second object in the thermal image, selecting the first object, and the generated callout indicates the first object, and the second object has no associated callout in the composite image.

[0162] The method may be implemented such that the callout includes a highlight or a shading of the detected object, an outline the detected object, a label identifying the detected object, or an alert.

[0163] A self-contained breathing apparatus (SCBA) is presented that includes a respirator, the respirator including an in-mask display component configured to display a graphical user interface to a wearer of the respirator, a thermal imaging system configured to capture a thermal image of a field of view of a scene in front of the wearer and an image processing system configured to: detect an object in the thermal image, the object is in front of the wearer, generate a callout for the object, and generate a composite image including the generated callout and the thermal image. The SCBA also includes a graphical user interface generator configured to generate the user interface for the in-mask display component, the generated user interface includes the composite image.

[0164] The system may be implemented such that the imaging processing system is physically remote from the SCBA, and a communication component communicates the generated graphical user interface to the in-mask display component using a wireless communication protocol. The system may be implemented such that the thermal imaging system includes a camera, and the camera is mounted to the SCBA.

[0165] The system may be implemented such that the image processing system is configured to detect the object using an object detector configured to, using a database of objects, identify the detected object.

[0166] The system may be implemented such that the image processing system is configured to detect edges in the thermal image, using an edge detection algorithm, and the object detector detects the image based on the detected edges identifies the detected object by comparing the detected edges to the database of objects.

[0167] The system may be implemented such that the object detector includes a trained machinelearning algorithm.

[0168] The system may be implemented such that the detected object is a first detected object, and the image processing system is further configured to: detect a second object in the thermal image, select the first object, and the generated callout indicates the first object, and the second object is not called out in the composite image.

[0169] The system may be implemented such that the callout includes: a highlight or a shading of the detected object, an outline the detected object, a label identifying the detected object, or an alert.

[0170] The system may be implemented such that the object detector is configured to identify the detected object based on a detected thermal gradient between the object and its surroundings.

[0171] The system may be implemented such that the image processing system includes a data receiver that receives a sensor signal from a sensor other than the thermal imaging system.

[0172] The thermal imaging system may be implemented such that the in-mask display unit includes a monocular eyepiece inside which the in-mask micro-display is mounted, and, the in-mask display unit further includes an objective lens and an ocular lens that are mounted in the monocular eyepiece rearward of the in-mask micro-display and that combine with the in-mask micro-display to form an optical pathway to deliver images to a single eye of the user.

Claims

What is claimed is:

1. A thermal imaging system configured for use with a respirator mask, comprising: an in-mask display unit configured to be mounted within an interior volume of a respirator mask and comprising an in-mask micro-display; a wearable thermal imaging camera that is configured to capture a thermal image and to communicate at least a portion of the captured thermal image to the in-mask microdisplay; and an image processing system comprising a processor and memory, the system being configured to: detect an object in the thermal image; generate a callout to indicate the detected object; compile a composite image, wherein the composite image comprises the thermal image and the generated callout.

2. The system of claim 1, wherein the image processing system is configured to detect the object using an object detector configured to, using a database of objects, identify the detected object.

3. The system of claim 2, wherein the image processing system is configured to detect edges in the thermal image, using an edge detection algorithm, and wherein the object detector detects the image based on the detected edges identifies the detected object by comparing the detected edges to the database of objects.

4. The system of claim 2, wherein the object detector comprises a trained machine-learning algorithm.

5. The system of any of claims 1-4, wherein the detected object is a first detected object, and wherein the image processing system is further configured to: detect a second object in the thermal image; select the first object; and wherein the generated callout indicates the first object, and wherein the second object is not called out in the composite image.

6. The system of any of claims 1-5, wherein the callout comprises: a highlight or a shading of the detected object; an outline the detected object; a label identifying the detected object; an overlay applied over the image; or an alert.

7. The system of claim 2, wherein the object detector is configured to identify the detected object based on a detected thermal gradient between the object and its surroundings.

8. The system of any of claims 1-7, wherein the image processing system comprises a data receiver that receives a sensor signal from a sensor other than the thermal imaging system.

9. The system of any of claims 1-8, wherein the display unit is a micro-display, the system further comprising: a control system for controlling the in-mask micro-display and the wearable thermal imaging camera, wherein the wearable thermal imaging camera comprises a wide field of view so that the captured thermal image comprises at least a primary subimage and a secondary subimage; and, wherein the control system is configured so that the in-mask microdisplay presents a default view in which the in-mask micro-display shows the primary subimage of the captured thermal image but does not show the secondary subimage of the captured thermal image; and wherein upon the control system receiving an input to shift the in-mask micro-display from the default view to an auxiliary view, the control system causes at least a portion of the primary subimage to be directionally displaced on the in-mask micro-display by the secondary subimage so that the secondary subimage is now shown in place of the displaced portion of the primary subimage.

10. The thermal imaging system of any of claims 1-9, wherein the in-mask display unit comprises a monocular eyepiece inside which the in-mask micro-display is mounted; and, wherein the in-mask display unit further comprises an objective lens and an ocular lens that are mounted in the monocular eyepiece rearward of the in-mask micro-display and that combine with the in-mask micro-display to form an optical pathway to deliver images to a single eye of the user.

11. A respirator mask comprising the thermal imaging system of any of claims 1-10, wherein the respirator mask defines an interior volume when fitted to the face of a human user.

12. The respirator mask of claim 11, wherein the wearable thermal imaging camera is configured to be attached to the outside of the respirator mask and is wirelessly connected to the control system.

13. The respirator mask of claim 11, wherein the wearable thermal imaging camera is configured to be attached to a protective helmet or a protective garment that is worn in conjunction with the respirator mask, and wherein the wearable thermal imaging camera is wirelessly connected to the control system.

14. A method of providing thermal images to a wearer of a respirator mask, the method comprising: capturing a thermal image using a thermal imaging system; processing the thermal image, using an image processing system, wherein processing comprises: detecting an object in the thermal image; generating an object callout for the detected object; generating a composite image comprising the thermal image and the detected object; and generating a graphical user interface for a display component of the respirator mask, the graphical user interface comprising the composite image; displaying the graphical user interface on a display coupled to the respirator mask.

15. The method of claim 14, wherein the thermal imaging system comprises a thermal imaging camera mounted to a self-contained breathing apparatus that comprises the respirator mask.

16. The method of claim 14 or 15, wherein detecting comprises an object detector comparing the thermal image to a datastore of reference objects.

17. The method of claim 16, wherein processing the thermal image further comprises detecting an edge, using an edge detection algorithm, and wherein the object detector detects the image based in part on the detected edge.

18. The method of claim 16, wherein the object detector comprises a trained machine learning algorithm.

19. The method of claim 16, wherein the method further comprises, based on the comparison, identifying the object.

20. The method of claim 19, wherein the method further comprises: detecting a thermal gradient between the detected object and its surroundings in the thermal image; and wherein the object is identified based on the thermal gradient.

21. The method of any of claims 14-20, wherein the detected object is a first detected object, and wherein the method further comprises: detecting a second object in the thermal image; selecting the first object; and wherein the generated callout indicates the first object, and wherein the second object has no associated callout in the composite image.

22. The method of any of claims 14-21, wherein the callout comprises: a highlight or a shading of the detected object; an outline the detected object; a label identifying the detected object; or an alert.

23. A self-contained breathing apparatus (SCBA) comprising: a respirator, the respirator comprising an in-mask display component configured to display a graphical user interface to a wearer of the respirator; a thermal imaging system configured to capture a thermal image of a field of view of a scene in front of the wearer; an image processing system configured to: detect an object in the thermal image, wherein the object is in front of the wearer; generate a callout for the object; and generate a composite image comprising the generated callout and the thermal image; anda graphical user interface generator configured to generate the user interface for the in-mask display component, wherein the generated user interface comprises the composite image.

24. The system of claim 23, wherein the imaging processing system is physically remote from the SCBA, and wherein a communication component communicates the generated graphical user interface to the in-mask display component using a wireless communication protocol.

25. The system of claim 23, wherein the thermal imaging system comprises a camera, and wherein the camera is mounted to the SCBA.

26. The system of claim 23, wherein the image processing system is configured to detect the object using an object detector configured to, using a database of objects, identify the detected object.

27. The system of claim 26, wherein the image processing system is configured to detect edges in the thermal image, using an edge detection algorithm, and wherein the object detector detects the image based on the detected edges identifies the detected object by comparing the detected edges to the database of objects.

28. The system of claim 26, wherein the object detector comprises a trained machine-learning algorithm.

29. The system of any of claims 23-28, wherein the detected object is a first detected object, and wherein the image processing system is further configured to: detect a second object in the thermal image; select the first object; and wherein the generated callout indicates the first object, and wherein the second object is not called out in the composite image.

30. The system of any of claims 23-29, wherein the callout comprises: a highlight or a shading of the detected object; an outline the detected object; a label identifying the detected object; or an alert.

31. The system of claim 26, wherein the object detector is configured to identify the detected object based on a detected thermal gradient between the object and its surroundings.

32. The system of any of claims 23-31, wherein the image processing system comprises a data receiver that receives a sensor signal from a sensor other than the thermal imaging system.

33. The system of any of claims 23-32, wherein the in-mask display unit comprises a monocular eyepiece inside which the in-mask micro-display is mounted; and, wherein the in-mask display unit further comprises an objective lens and an ocular lens that are mounted in the monocular eyepiece rearward of the in-mask micro-display and that combine with the in-mask micro-display to form an optical pathway to deliver images to a single eye of the user.

Images