Image Processing Device

Abstract

To provide an image processing technique for processing image data obtained from visible and thermography cameras sharing at least a partial field of view.SOLUTION: The method includes acquiring two images showing the same real world scene, in which a first image contains a visible light image and a second image contains a thermal image, identifying one or more spatial parts of the second image representing a predetermined temperature range, identifying one or more spatial parts of the first image representing the same part of the real world scene as shown in one or more identified spatial parts of the second image, and deriving a composite image based on the first image to emphasize one or more identified spatial parts of the first image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to processing respective image data obtained from a visible light camera and a thermographic camera that at least partially share a field of view. [Background technology]

[0002] For example, many detection and surveillance methods rely at least in part on visual information obtained using camera assemblies that include one or more camera modules or units to capture still images and / or video streams of a location of interest. Such methods include portable devices that can be brought to a site of use for direct visual inspection or for capturing images / video depicting a location of interest that is inaccessible to fixed installations for constant monitoring of the area or space of interest.

[0003] A specific example of the use of such a camera assembly is the so-called search camera device, such as the Savox SearchCam3000. These devices include a camera assembly located at one end of an elongated shaft and a display unit located at or near the other end of the shaft. The display unit is configured to display the current camera field of view to a user of the search camera, enabling the user to, for example, locate victims or objects of interest in a collapsed building, locate locations or objects underwater, etc. Some search camera devices are switchable between normal and infrared camera operation to allow the user to utilize the portion of the electromagnetic spectrum that provides the most useful visual information under the current operating conditions. Summary of the Invention [Problem to be solved by the invention]

[0004] The object of the present invention is to provide a technique that makes it possible to obtain enhanced visual information in order to improve the detection of objects of interest, for example when using a surveillance camera device of the type described above. [Means for solving the problem]

[0005] The following presents a simplified summary of some embodiments of the invention in order to facilitate a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention, and is not intended to identify key or critical elements of the invention or to delineate the scope of the invention. The following summary is intended merely to present some concepts of the invention as a prelude to a more detailed description of exemplary embodiments of the invention.

[0006] According to an exemplary embodiment, an image processing device is provided, the device comprising at least one processor and at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to cause the device, using the at least one processor, to perform the following: acquire a first image and a second image at least partially depicting a same real-world scene, where the first image is a visible light image and the second image is a thermal image; identify one or more spatial portions of the second image that represent a predetermined temperature range; identify one or more spatial portions of the first image that represent the same portion of the real-world scene as shown in the identified one or more spatial portions of the second image; and derive a composite image based on the first image that enhances the identified one or more spatial portions of the first image.

[0007] According to another exemplary embodiment, an imaging device is provided, comprising a camera assembly consisting of a first imaging device for capturing visible light images and a second imaging device for capturing thermal images, the first and second cameras positioned relative to each other such that their respective fields of view at least partially overlap, and an image processing device as described in the above exemplary embodiment.

[0008] According to another exemplary embodiment, there is provided a surveillance camera apparatus comprising an elongate shaft, a display, and an imaging apparatus as described in the previous exemplary embodiment, where the camera assembly is disposed at one end of the elongate shaft and the display is disposed at or near an opposite end of the elongate shaft, and the image processor is further adapted to provide a composite image for display on the display.

[0009] According to another exemplary embodiment, a method for image processing is provided, the method including: acquiring a first image and a second image showing a same real-world scene, the first image including a visible light image and the second image including a thermal image, identifying one or more spatial portions of the second image that represent a predetermined temperature range, identifying one or more spatial portions of the first image that represent the same portion of the real-world scene as shown in the identified one or more spatial portions of the second image, and deriving a composite image based on the first image, enhancing the identified one or more spatial portions of the first image.

[0010] According to another exemplary embodiment, a computer program is provided comprising computer readable program code configured to cause at least the method according to the aforementioned exemplary embodiment to be performed when the program code is executed on one or more computing devices.

[0011] A computer program according to an exemplary embodiment may be embodied on a volatile or non-volatile computer readable storage medium, for example as a computer program product comprising at least one computer readable non-transitory medium having program code stored thereon, which when executed on one or more devices causes the one or more devices to perform at least the operations described above for the computer program according to an exemplary embodiment of the present invention.

[0012] The exemplary embodiments of the invention described herein should not be construed as limiting the applicability of the appended claims. The verbs "comprise" and "include" and their derivatives are used herein as open limitations that do not exclude the presence of unrecited features. It is further to be understood that the use of "a" or "an" herein, i.e., the singular, does not exclude a plurality. Moreover, the features described above can be freely combined with each other unless otherwise specified.

[0013] Some features of the present invention are set forth in the appended claims, however the inventive aspects, both as to its structure and method of operation, together with additional objects and advantages thereof, will be best understood from the following description of several illustrative embodiments read in connection with the accompanying drawings.

[0014] In the following, non-limiting exemplary embodiments of the present invention and some of its advantages are explained in detail with reference to the accompanying drawings. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a block diagram of some logical elements of an imaging device according to one embodiment. [Diagram 2] FIG. 2 is a block diagram of some logical elements of an image processing apparatus according to one embodiment. [Diagram 3] FIG. 2 illustrates a method according to an embodiment. [Figure 4] FIG. 2 is a schematic diagram of a survey camera according to one embodiment. [Diagram 5] FIG. 2 is a block diagram of some elements of an apparatus according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] 1 is a block diagram of some logical elements of an imaging device 100 according to one embodiment. The imaging device 100 comprises a camera assembly 110 that is communicatively coupled to an image processor 120 via a communications link 130. In this regard, the camera assembly 110 comprises a first camera 111 and a second camera 112, each of which is communicatively coupled to the image processor 120. In other embodiments, the imaging device 100 may include additional components and / or some of the components shown in FIG. 1 may be combined with other components.

[0017] The communicative coupling may allow images captured by the first camera 111 and the second camera 112 to be transferred from the camera assembly 110 to the image processor 120, and may also allow control information to be transferred from the image processor 120 to the camera assembly 110. Additionally, the communicative coupling may allow other information to be transferred between the camera assembly 110 and the image processor 120. The communicative coupling may be provided, for example, using a wired or wireless communication link between the camera assembly 110 and the image processor 120. As one example in this regard, the coupling may be provided via one or more electrical wires and / or one or more optical links, such as optical fibers. In another example, the communicative coupling may be provided using a short-range wireless communication technology known in the art, such as a wireless local area network conforming to the IEEE 802.11 series of standards or a personal area network conforming to the Bluetooth standard. As a further example, the communicative coupling may be provided using a cellular communication technology known in the art.

[0018] The image processing device 120 may be configured to synthesize an image based on respective image data received from the first camera 111 and / or the second camera 112. Thus, the image synthesized by the image processing device 120 may be an image captured by the first camera 111 or a derivative thereof, an image captured by the second camera 112 or a derivative thereof, or a synthetic image synthesized based on respective time-aligned images captured by the first camera 111 and the second camera 112. In the following, for clarity and brevity of explanation, the term synthetic image is applied to an image derived by the image processing device 120 for presentation to a user, which in some scenarios may not be a true synthetic image but an image derived based on an image captured by the first camera 111 or an image captured by the second camera 112.

[0019] FIG. 1 is intended to illustrate logical components of the imaging device 100 and is not intended to impose architectural constraints on the implementation of the imaging device 100. As an example, the first camera 111 and the second camera 112 may both be provided as separate components included in the camera assembly 110, or in another example, the first camera 111 and the second camera 112 may be provided as a single component, where the first camera 111 and the second camera 112 are separate subcomponents or the first camera 111 and the second camera 112 share some parts within the single component to provide the first and second cameras 111, 112. In another example, the image processor 120 may be provided as one or more components configured to provide (possibly jointly) the functionality described above. In this regard, the image processor 120 may be provided as separate hardware or a combination of hardware and software configured to implement the functionality described above for the image processor 120.

[0020] The first camera 111 may be for capturing images using visible light and the second camera 112 may be for capturing images using infrared (IR) radiation. The first camera 111 using the visible portion of the electromagnetic radiation spectrum may be referred to as, for example, a visible light camera or a normal camera, and the second camera 112 using the IR portion of the electromagnetic radiation spectrum may be referred to as, for example, an infrared camera (IR camera) or a thermographic camera or a thermal imaging camera. The first camera 111 may be configured to capture visible light images based on a first wavelength range of the received electromagnetic radiation and the second camera 112 may be configured to capture thermal images based on a second wavelength range of the received electromagnetic radiation, where the second range includes longer wavelengths than the first range. Typically, the first range covers wavelengths from about 400 to 700 nanometers (nm) and the second range includes wavelengths above 700 nm, for example from about 700 to 1000 nm. In some embodiments, the second range may extend to even longer wavelengths, such as wavelengths in the microwave range (eg, thousands of nanometers).

[0021] The first camera 111 may be, for example, a digital camera unit or module configured to capture color images. In another example, the first camera 111 may be a digital camera unit or module configured to capture monochrome images, such as grayscale images. In a further example, the first camera 111 may be a digital camera unit or module configured to selectively capture one of a color image or a monochrome image according to control information received from the image processing device 120. While a color image generally conveys a more accurate representation of the scene represented in the image, a monochrome image may provide better image quality, especially under low lighting conditions. Therefore, being able to select a device utilizing a first camera 111 of a desired type (e.g., color image or monochrome image) or being able to select an operating mode for the first camera 111 (e.g., color image or monochrome image) may be advantageous under certain operating conditions. Instead of the first range mentioned previously, a monochrome image may be obtained, for example, by using the short wavelength IR range (approximately 900 to 1700 nm).

[0022] Cameras suitable for use as the first and second cameras 111, 112 are well known in the art, and any suitable individual camera units or camera modules may be employed as the first camera 111 and the second camera 112. Accordingly, details of the structure and / or operation of each of the first camera 111 and the second camera 112 are not provided in this disclosure, as they are readily available to those skilled in the art, apart from a functional perspective relevant to the description of certain aspects of their usage as part of the camera assembly 110.

[0023] The first camera 111 and the second camera 112 are arranged in fixed positions relative to each other in the camera assembly 110 such that their respective fields of view at least partially overlap. It is advantageous for the overlap to be as wide as possible, in the sense that the first camera 111 and the second camera 112 can capture respective images that represent substantially the same scene and therefore substantially the same real-world object. As an example, this is achieved by arranging the first camera 111 and the second camera 112 relatively close to each other such that a first optical axis 113 of the optical system of the first camera 111 is parallel or substantially parallel to a second optical axis 114 of the optical system of the second camera 112 at a predetermined distance D therefrom. The concept of "relatively close" depends on the physical size of the camera assembly 110, the physical size of the first camera 111 and the second camera 112, and the desired or typical distance from the camera assembly 110 to the real-world object of interest. In one example, the first camera 111 and the second camera 112 are positioned side-by-side as close to each other as possible to ensure maximum overlap between the respective fields of view of the first camera 111 and the second camera 112. As a result, by way of example, the distance D between the first optical axis 113 and the second optical axis 114 may be a few centimeters.

[0024] In line with the above description, the camera assembly 110 may be operated under the control of the image processor 120 in accordance with control information transferred via the communicative coupling. Respective images captured by the first camera 111 and the second camera 112 may be transferred from the camera assembly to the image processor 120 for analysis and / or processing. In particular, the camera assembly 110 may be configured to capture and provide a single (still) image from each of the first and second cameras 111, 112, or may be configured to capture and provide a respective sequence of images (e.g., respective video streams) from each of the first and second cameras 111, 112.

[0025] As an example, the control information transferred from the image processing device 120 to the camera assembly 110 may include instructions or commands to start or stop capturing image streams (e.g., video streams) using the first and second cameras 111, 112 and providing these streams to the image processing device 120. The instructions or commands to start capturing and providing the image streams may include an indication of a frame rate to be applied to the image streams provided from the camera assembly 110 to the image processing device 120. In one example, the frame rate may be a suitable value selected from the range of 5 to 240 frames per second (fps), for example 30 fps. In another example, the camera assembly 110 may be configured to use a predetermined fixed frame rate (e.g., 30 fps), and therefore the instructions or commands to start capturing and providing the image streams do not need to indicate an applicable frame rate.

[0026] In the above example, the applied frame rate is implicitly assumed to be the same for the first camera 111 and the second camera 112. In other examples, different frame rates may be used for the first camera 111 and the second camera 112, with the respective frame rates for the first camera 111 and the second camera 112 being selected or (pre)set independently of each other.

[0027] As another example, the control information may include instructions or commands to capture respective (still) images using the first camera 111 and the second camera 112 and provide the resulting image pairs to the image processor 120 .

[0028] As a further example, the control information may include individual instructions or commands to appropriately adjust or set one of a number of camera parameters, such as image size, image resolution, focal length of the camera optics, etc. The control information may allow such parameters of the first camera 111 and the second camera 112 to be adjustable independently of one another, or the adjustment of the camera parameters may be simultaneous and similar for each of the first camera 111 and the second camera 112.

[0029] In yet another example, the control information may include an indication of whether the first camera 111 is to be operated in a color or monochrome image mode, with the possibility of selecting between monochrome and color when using the first camera 111. The monochrome image mode (which may, as discussed above, rely for example on the short wavelength IR spectrum instead of the visible light spectrum) may be useful for providing images of improved perceived quality in low light conditions.

[0030] In yet another example, the control information may include instructions or commands to enable or disable operation of the first camera 111 and / or the second camera 112. If only one of the first and second cameras 111, 112 is enabled, operation of the image processor 120 will result in a composite output image based on images from the enabled one of the first and second cameras 111, 112. However, a true composite image will result from operation of the image processor 120 in a scenario where both the first camera 111 and the second camera 112 are enabled simultaneously or substantially simultaneously.

[0031] In yet another example, the camera assembly 110 further comprises at least one light source for illuminating the field of view of the first and second cameras 111, 112 to enable capture of usable images in poorly lit conditions or dark spaces. In such a configuration, the control information can include instructions or commands to turn on or off the light output from the at least one light source.

[0032] Exemplary instructions or commands for operating one or both of the first and second cameras 111, 112 and / or other components of the camera assembly 110 may be provided based on user input received via a user interface (UI) of a device utilizing the image processing device 100.

[0033] Above and below, the components configured to capture respective images using visible light and IR radiation are referred to as the first camera 111 and the second camera 112, respectively. However, this is a choice for editorial clarity and descriptive brevity, and the present disclosure is not limited to implementations utilizing elements specifically designated as separate camera units or camera modules, and / or methods utilizing components specifically designated as visible light images or thermal images. In this regard, the first camera 111 is generalized as a first image device comprising a sensor array sensitive to visible light (e.g. an image sensor array provided as a semiconductor charge-coupled device (CCD), or a CMOS sensor array), capable of capturing an array of pixel values ​​describing the intensity (and possibly also the color) of the visible light received at the sensor array at each spatial location. Similarly, the second camera 112 is generalized as a second image device including an array of temperature-sensitive sensors (e.g. an array of IR sensors), capable of capturing an array of pixel values ​​describing the temperature sensed at each spatial location of the sensor array. Each of the first and second imaging devices includes further elements of imaging devices known in the art, such as a shutter assembly to enable proper exposure of the sensor array and an optical system capable of focusing incident electromagnetic radiation towards the sensor array.

[0034] 2 is a block diagram of some logical elements of an image processor 120 according to one embodiment. The image processor 120 comprises an image analyzer 121 and an image combiner 122. In other embodiments, the image processor 120 includes additional components and / or the image analyzer 121 and image combiner 122 are combined with each other and / or with one or more other components.

[0035] The image analyzer 121 receives respective images from the first camera 111 and the second camera 112 of the camera assembly 110 via the communication link 130 and performs image analysis as described below by way of non-limiting examples. The output of the image analysis is provided to the image combiner 122, which combines a composite image based on the images captured by at least one of the first and second cameras 111, 112 and the output of the image analysis. The image combiner 122 further sends the composite image for display via the UI of a device utilizing the image processing device 100.

[0036] The image analyzer 121 and the image combiner 122 may be configured to jointly perform the processes described herein as steps of a method 200 illustrated by the flow chart shown in Fig. 3. In the following, the method 200 is described with respect to the derivation of a composite image based on a pair of time-aligned images, a first image captured using a first camera 111 of the camera assembly 110 and a second image captured using a second camera 112 of the camera assembly 110. However, the method 200 may be easily generalized to derive a series of composite images (e.g., a video stream) based on a series of time-aligned pairs, each of which includes a series of first images captured using the first camera 111 of the camera assembly 110 and a series of second images captured using the second camera 112 of the camera assembly 110. The method 200 may be modified in a number of ways, for example following the examples of the functions of the components of the image processing device 120 described above and below.

[0037] The method 200 begins with acquiring a first image and a second image, as shown in block 202. As previously described, the first image includes an image captured by the first camera 111, and the second image includes an image captured by the second camera 112. As previously described, the image processor 120 receives one or more first images (e.g., a series of first images) from the first camera 111 via the communication link 130 and stores the received images in a memory provided in the image processor 120. Similarly, the image processor 120 receives one or more second images (e.g., a series of second images) from the second camera 112 via the communication link 130 and stores the received images in a memory provided in the image processor 120.

[0038] As an example in this regard, the image processor 120 may be configured to store and retain in memory a predetermined number of the most recently captured first and second images, thereby deleting the oldest image from memory each time a new image is received from the camera assembly 120. In one example, the predetermined number may be one, such that the image processor memory stores only the most recently captured first image and the most recently captured second image. In another example, the image processor 120 may be configured to store and retain in memory the most recently captured images for a predetermined length of time (e.g., a period selected from a range of a fraction of a second to several seconds). As a further example, the image processor 120 may be configured to store and retain in memory all first and second images received from the camera assembly 110 (e.g., until deleted in response to a user action or command).

[0039] According to one example, the first and second images processed as a pair by the image processor 120 are captured simultaneously or substantially simultaneously. According to another example, a first image is "paired" with the image in the series having the closest capture time to the capture time of the first image. Or vice versa, a second image is "paired" with the image in the series having the closest capture time to the capture time of the first image. As a result, time-aligned first and second images are obtained.

[0040] Due to the aforementioned configuration of the first and second cameras 111, 112 (i.e. with their respective fields of view partially overlapping with respect to the parallel optical axes 113, 114 at a distance D), the first and second time-aligned images represent at least partially the same real-world scene. As a result, the first and second images have overlapping portions in both images that represent the same real-world scene at the locations where the first and second images were captured. However, due to different characteristics of the images, i.e. the first image includes a visible light image depicting a scene and the second image includes a thermal image depicting at least partially the same scene, the second image provides information that is complementary to the information provided by the first image (and vice versa).

[0041] Method 200 further identifies one or more spatial portions of the second image that represent a predetermined temperature range, as shown in block 204. Because the second image is a thermal image and the pixel values ​​of the second image are directly proportional to the temperature of the real-world object depicted therein, the second image is capable of locating the depicted object by its temperature. As used herein, identifying one or more spatial portions that represent a predetermined temperature range may include, for example, identifying one or more sets of pixel locations of the second image where the pixel values ​​of the second image are in a predetermined range of pixel values ​​that indicate the predetermined temperature range, thereby identifying the depicted real-world object having a temperature within the predetermined temperature range.

[0042] In one example, the predefined temperature range is provided as a range of preset temperature values ​​during manufacture and calibration of the imaging device 100. In another example, the predefined temperature range is selected based on user input received via a user interface (UI) configured to control at least some aspects of the operation of the imaging device 100. In this regard, for example, the (ends of the) predefined temperature range may be set based on user input received via the UI, or the (ends of the) predefined temperature range may be selected from a plurality of predefined temperature ranges based on user input received via the UI.

[0043] The predetermined temperature range of interest depends on the application of the imaging device 100. Non-limiting examples of the predetermined temperature range of interest include the temperature range of a living human body, such as 30-50°C, 15-44°C, or any other temperature range that may be suitable for the intended use of the imaging device 100. This may be useful for detection and surveillance applications, for example, aiming to identify people depicted in the first and second images. Another example of a temperature range of interest includes the range of combustion temperatures of different materials typically found in buildings, such as a range of 300-1025°C.

[0044] The mapping between the predefined temperature range and the corresponding range of pixel values ​​can be given by a predefined temperature mapping function, which may be defined via a calibration procedure during the manufacture of a device utilizing the imaging device 100 and / or when the imaging device 100 is configured or reconfigured (e.g., by a user or maintenance personnel). As an example, the calibration procedure may include defining a mapping between pixel values ​​and corresponding temperatures for a number of temperatures within a range between a predefined minimum temperature and a predefined maximum temperature, where the predefined temperature range of interest is within the entire range bounded by the minimum and maximum temperatures (i.e., the configured range of temperatures includes temperatures below and / or above the temperature range of interest). In another example, the calibration procedure includes defining a mapping between pixel values ​​and corresponding temperatures for a number of temperatures within a range from a lower limit of the temperature range of interest to an upper limit of the temperature range of interest. Thus, the calibration only includes the temperature range of interest. In a further example, the calibration procedure includes defining a mapping between pixel values ​​and corresponding temperatures at a lower limit of the temperature range of interest and an upper limit of the temperature range of interest. This calibration therefore only includes the boundaries of the temperature range of interest.

[0045] The method further includes identifying one or more spatial portions of the first image that represent portions of the real-world scene that are the same as those depicted in the identified one or more spatial portions of the second image, as shown in block 206. Thus, the identified one or more spatial portions of the first image serve to depict the same real-world objects as depicted in the identified one or more spatial portions of the second image. In this regard, because the spatial overlap of the first and second images is not complete, it is possible that the portions of the real-world scene depicted in some of the identified one or more spatial portions of the second image may not appear at all in the first image or may only appear partially within the image.

[0046] In this example, the first and second images are captured using a first camera 111 and a second camera 112, respectively, with respective optical axes 113 and 114 parallel to each other at a distance D, so that there is a simple mapping relationship between pixel locations in the second image, which represent a particular point of the real-world scene, and corresponding pixel locations in the first image, which also represent a particular point of the real-world scene. This relationship between pixel locations in the second image can be defined through a spatial mapping function, which in this example includes a translation along the axis of the pixel location. As an example, if the optical axes 113 and 114 are spaced apart from each other by a distance D along an axis x, then the spatial mapping from the pixel location in the second image to the corresponding pixel location in the first image includes a translation or movement along the axis x. Further considering a pixel at a location (x2, y2) in the image plane of the second image, the corresponding location (x1, y1) in the first image can be found, for example, using the spatial mapping function (x1, y1)=(x2+D, y2).

[0047] In the above example, the first camera 111 and the second camera 112 are assumed to be arranged side-by-side, so that pixel positions representing the same point in the real-world scene are translated along the horizontal axis of the image plane. Such mapping of pixel positions is easily generalized for any (predetermined) relative positions between the first and second cameras 111, 112. Thus, spatially mapping a pixel position in the second image to a corresponding spatial position in the first image is generalized to mapping a pixel position in the second image to a corresponding pixel position in the first image by translating the pixel position in the second image by a given amount in a given direction. The general form of the spatial mapping function is, for example, (x1, y1)=(x2+d x ,y2+d y ), where (d x ) 2 +(d y ) 2 =D 2 It is.

[0048] The spatial mapping function may be defined during the manufacture of a device utilizing the imaging device 100 and / or during the configuration or reconfiguration of the imaging device 100. As a non-limiting example in this regard, the first and second cameras 111, 112 may be used to capture respective images of a scene containing a point in a uniform background, and a translation (e.g., d as defined above) may be used to align the point depicted in the second image with the location depicted in the first image. x and d y ), it can be found by finding

[0049] Thus, identifying one or more spatial portions of the first image depicting the same portion of the real-world scene depicted in the identified one or more spatial portions of the second image includes locating the one or more spatial portions of the first image based on the identified one or more spatial portions of the second image using a spatial mapping function. In this regard, the spatial mapping function may be applied pixel-by-pixel. For example, each set of pixel locations belonging to the set of one or more pixel locations defining the identified one or more spatial portions of the second image is transformed using the spatial mapping function into a corresponding set of pixel locations of the first image, thereby obtaining a corresponding set of one or more pixel locations of the first image that defines the identified one or more spatial portions of the first image.

[0050] The method 200 further includes deriving a composite image based on the first image, as shown at block 208, in which the identified spatial portion or portions of the first image are enhanced. As previously described, the identified spatial portion or portions of the first image may be defined as a set of one or more pixel locations of the first image, respectively. As an example in this regard, deriving the composite image based on the first image may include generating a composite image in which an enhanced image is superimposed on (a copy of) the first image, where the enhanced image includes respective visual indications at spatial locations that correspond spatially to the identified spatial portion or portions of the first image, and portions outside the enhanced image are transparent. As a result, the visual indications introduced via the enhanced image serve to highlight real-world objects depicted in the first image that are found to represent a predetermined temperature range in the second image. In another example, the identified spatial portion or portions of the first image may be directly modified to include respective visual indications, such as by replacing image content of the identified spatial portion or portions of the first image with the respective visual indications or by modifying image content of the identified spatial portion or portions of the first image in a predetermined manner.

[0051] In one example, the visual indication introduced into the composite image, e.g., by an enhanced image or by replacing each identified spatial portion of the first image with a corresponding image content, may include introducing the respective image content as a predetermined color and / or a predetermined pattern. In another example, the visual indication introduced into the composite image, e.g., by modification of each identified spatial portion of the first image, may include adding a predetermined value to each pixel value within each identified spatial portion of the first image or subtracting a predetermined value from each pixel value within each identified spatial portion of the first image to enhance each identified spatial portion of the first image, while preserving at least some of its original display characteristics imparted to the first image.

[0052] In a further example, the visual indication introduced into the composite image, e.g., by the enhanced image or by replacing each identified spatial portion of the first image with a corresponding image content, may include outlining each identified spatial portion of the first image, e.g., with a line or border of a predetermined color and / or a predetermined pattern, following or substantially following the shape of the outline of each identified spatial portion of the first image. In a further example, the visual indication introduced into the composite image, e.g., by the enhanced image or by replacing the corresponding image content of the first image, may include modifying the image content around each identified spatial portion of the first image in a predetermined manner, e.g., by at least partially surrounding each identified spatial portion of the first image within a frame of a predetermined shape (e.g., a rectangular shape), where the frame may be provided using a predetermined color and / or a predetermined pattern. In yet another example, the visual indication introduced into the composite image, e.g., by replacing the corresponding image content of each identified spatial portion of the first image with the enhanced image or the corresponding image content, may include the image content of each identified spatial portion of the second image. Here, such a visual indication may be introduced by modification of each identified spatial portion of the first image, for example by adding image content of each identified spatial portion of the second image to each identified spatial portion of the first image, or by subtracting image content of each identified spatial portion of the second image from each identified spatial portion of the first image.

[0053] The method 200 may further include providing the composite image for display via a display, as shown in block 210. The display may be provided as part of a UI configured to control at least some operational aspects of the imaging device 100. The composite image thus provides a user of the imaging device 100 with a highlighting of real-world objects at the location of capture of the first and second images by highlighting real-world objects found to be in a temperature range of interest, thereby enabling improved and / or more efficient use of the imaging device 100, for example, in scenarios involving identifying real-world objects at or near the location that exhibit a temperature range of interest.

[0054] As a non-limiting example of a device that utilizes the imaging device 100, FIG. 4 shows a schematic representation of a surveillance camera device 300 according to one embodiment. The surveillance camera device 300 comprises an elongated shaft 302 having a camera assembly 110 at or near one end (e.g., a front or distal end) of the shaft 302, and a housing 304 mounted at or near the other end (e.g., a rear or proximal end) of the shaft 302, including an image processor 120 and a display (possibly together with one or more other components of a UI). A communications link 130 may be provided via one or more electrical wires and / or via one or more optical links, such as optical fibers provided within the shaft 302. The surveillance camera device 300 may be used, such as by a user positioning the front end of the shaft 302 such that the first and second cameras 111, 112 of the camera assembly 110 can view locations where direct observation is not possible or is dangerous, such as inside a collapsed building or underwater. After the exploration camera device 300 has been so positioned, the user activates the imaging device 100 therein to, for example, capture a composite (still) image or a series of composite images (e.g., a video stream) for viewing on a display provided in the housing 304.

[0055] In one variation of the surveillance camera device 300, instead of arranging the camera assembly 110 and the housing 304 containing the image processor 120 and the display on the shaft 302, the camera assembly 110 may be arranged on a first component and the housing 304 may be arranged on a second component, which may be movable relative to each other, thereby improving the freedom of positioning of the camera assembly 110 for observing the location of interest. In such a configuration, the first and second components may be coupled to each other, for example, via a flexible cable having a communication link 130 therein (e.g., one or more wires and / or one or more optical links, such as optical fibers). In another example, there is no physical link between the first and second components, and the communication link 130 is provided using a suitable wireless technology known in the art. Non-limiting examples in this regard include cellular and short-range wireless communication technologies known in the art, such as wireless local area networking (e.g., Wi-Fi) according to the IEEE 802.11 series of standards, or personal area networking according to the Bluetooth® standard. This allows greater freedom in positioning the camera assembly 110 for observing a location of interest.

[0056] In the above, the operation of the imaging device 100 and / or the method 200 is implicitly described with reference to identifying one or more spatial portions that represent one predetermined temperature range of interest, where spatial portions that represent other temperature ranges may or may not be ignored. According to an example, the imaging device 100 and / or the method 200 may be modified such that identifying one or more spatial portions of the second image (see block 204) involves identifying a respective set of one or more spatial portions of the second image, each of which represents a respective one of a plurality of different predetermined temperature ranges of interest. In this regard, the number of temperature ranges considered may be two, three, four, etc., or any number that is deemed applicable in view of the intended application of the imaging device 100 and / or the method 200. As a result, identifying spatial portions of the first image that depict the same part of the real-world scene as depicted in the identified spatial portions of the second image may be performed for each set of one or more spatial portions identified in the second image (see block 206). A respective set of one or more spatial portions of the first images is thereby derived for each temperature range of interest. Additionally, deriving the composite image (see block 208) may include using different emphasis of the identified spatial portions of the first images in each set, thereby providing a different visual indication for each of the multiple temperature ranges of interest (e.g., different colors, different patterns, and / or different color / pattern lines or borders surrounding the different sets of identified spatial portions, etc.).

[0057] That is, with reference to method 200, the above example having multiple temperature ranges of interest may include performing operations with respect to blocks 202 and 210 as described above in connection with method 200. Conversely, each of the operations with respect to blocks 202 through 210 may be repeated for each temperature range of interest to define each set of one or more spatial portions of the first image, and the operation with respect to block 208 may then be performed to provide a different visual emphasis for each set of one or more spatial portions, thereby providing a different visual emphasis for each temperature range of interest.

[0058] As one non-limiting example, such multiple temperature ranges of interest may be applicable, for example, in the case of a surveillance camera device 300 designed and / or configured to search for humans in high-risk locations such as inside a collapsed building, in a mining rescue scenario, etc. As a non-limiting example in this regard, the multiple temperature ranges may include two or more of the following: - A first temperature range of interest that represents normal human body temperature (e.g., 30-38°C) - A second temperature range of interest (e.g., 15-30°C) that represents temperatures below normal human body temperature. - A third temperature range of interest that represents temperatures higher than normal human body temperature (e.g., 38-44°C). A fourth temperature range of interest (e.g., 44-60°C) that represents objects that may be harmful to humans due to excessive heat when in direct contact or in close proximity for long periods of time. - A fifth temperature range of interest (e.g., 60°C or higher) representing objects that can be immediately harmful to humans due to excessive heat when in direct contact or in close proximity.

[0059] The above examples of applicable temperature ranges may enable techniques such as highlighting individuals with higher or lower than normal body temperatures in the composite image differently from individuals with normal body temperatures to help immediately identify and identify individuals who may require emergency medical attention, and / or highlighting hot objects that may be harmful to humans upon direct contact in the composite image differently than the highlighting applied to humans.

[0060] Figure 5 is a block diagram of some components of an exemplary device 500. The device 500 may include additional components, elements or portions not shown in Figure 5. The device 500 may be used to implement the image processing device 120, for example, using a combination of hardware and software.

[0061] The device 500 comprises a processor 516 and a memory 515 for storing data and computer program code 517. The memory 515 and the portions of the computer program code 517 stored therein together with the processor 516 may be further configured to implement at least some of the operations, procedures and / or functions described above in relation to the image processing device 120.

[0062] The device 500 includes a communication unit 512 for communicating with other devices. The communication unit 512 includes at least one communication device that allows wired or wireless communication with other devices. The communication devices of the communication unit 512 may be referred to as respective communication means.

[0063] The device 500 may further comprise a user I / O (input / output) component 518, which, possibly together with the processor 516 and a portion of the computer program code 517, may be configured to receive input from and / or provide output to a user of the device 500 to provide a user interface for controlling at least some aspects of the operation of the imaging device 120. The user I / O component 518 may comprise hardware components such as a display, a touch screen, a touch pad, a mouse, a joystick, a keyboard and / or one or more key or button arrangements. The user I / O component 518 may also be referred to as a peripheral device. The processor 516 may be configured to control the operation of the device 500, for example according to the portion of the computer program code 517, and possibly further according to user input received via the user I / O component 518 and / or according to information received via the communication unit 512.

[0064] Although the processor 516 is shown as a single component, it may be implemented as one or more separate processing components. Similarly, although the memory 515 is shown as a single component, it may be implemented as one or more separate components. Some or all of it may be integrated / removable storage and / or may provide persistent / semi-persistent / dynamic / cache storage.

[0065] The computer program code 517 stored in the memory 515 may include computer-executable instructions that, when loaded into the processor 516, control one or more aspects of the operation of the device 500. By way of example, the computer-executable instructions may be provided as one or more sequences of one or more instructions. The processor 516 may load and execute the computer program code 517 by reading from the memory 515 the one or more sequences of one or more instructions contained therein. The one or more sequences of one or more instructions, when executed by the processor 516, may be configured to cause the device 500 to perform at least some of the operations, procedures and / or functions described above with respect to the imaging device 120.

[0066] Thus, the device 500 comprises at least one processor 516 and at least one memory 515 containing computer program code 517 for one or more programs, which, together with the at least one processor 516, enable the device 500 to perform at least some of the operations, procedures and / or functions described above in relation to the image processing device 120.

[0067] The computer programs stored in the memory 515 may be provided as respective computer program products including at least one computer readable non-transitory medium having computer program code 517 embedded therein which, when executed by the device 500, causes the device 500 to perform at least some of the operations, procedures and / or functions described above with respect to the imaging device 120. The computer readable non-transitory medium may comprise a memory device or recording medium such as a CD-ROM, a DVD, a Blu-ray disc or another article tangibly embodying a computer program. As another example, the computer program may be provided as a signal configured to reliably transmit the computer program.

[0068] References to a processor should not be understood to encompass only programmable processors, but also special purpose circuitry such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), signal processors, etc. Features described in the above description may be used in combinations other than those explicitly stated.

[0069] The specific examples provided in the above description should not be considered limiting. Thus, the present invention is not limited to only the embodiments described above. Features described in the above description may be used in combinations other than those expressly described. Although functions have been described with reference to particular features, these functions may be performed by other features, whether or not described. Although features have been described with reference to particular embodiments, these features may also be present in other embodiments, whether or not described.

Claims

1. A surveillance camera device for detecting humans in high-risk locations, comprising an elongated shaft, a display, and an imaging device including a camera assembly and an image processing device, the camera assembly is disposed at one end of the elongated shaft; the display is disposed at or near an opposite end of the elongate shaft; the camera assembly includes a first imaging device that captures visible light images and a second imaging device that captures thermal images, the first imaging device and the second imaging device being positioned relative to one another such that their respective fields of view at least partially overlap; the image processor configured to receive a first image and a second image from the first imaging device and the second imaging device, respectively; the image processing device comprising at least one processor and at least one memory containing computer program code for one or more programs; The at least one memory and the computer program code include: acquiring the first image and the second image, the first image being a visible light image and the second image being a thermal image, the first image and the second image at least partially depicting a same real world scene; automatically identifying a corresponding set of one or more spatial portions of the second image that represent a corresponding one of a plurality of different predetermined temperature ranges including a first predetermined temperature range representing a human body temperature and a second predetermined temperature range representing temperatures of objects that may be harmful to humans due to excessive heat if in direct contact or in close proximity thereto, by covering temperatures of 44° C. or greater; for each set of one or more spatial portions of the second image, identifying a corresponding set of one or more spatial portions of the first image that illustrates an identical portion of a real-world scene as shown in the corresponding set of the identified one or more spatial portions of the second image; deriving a composite image based on the first image, the composite image having a different emphasis applied to each of the set of one or more spatial portions of the first image by enhancing the identified set of one or more spatial portions of the first image, and providing the composite image to the display for display; and causing the image processing device to execute, using the at least one processor, Exploration camera equipment.

2. A surveillance camera device for detecting humans in high-risk locations, the surveillance camera device comprising a display and an imaging device including a camera assembly and an image processing device, the surveillance camera device comprises a first entity having the camera assembly and a second entity having a housing containing the image processor and the display, the second entity being communicatively coupled to the first entity via a communications link, the first entity being movable relative to the second entity; the camera assembly includes a first imaging device that captures visible light images and a second imaging device that captures thermal images, the first imaging device and the second imaging device being positioned relative to one another such that their respective fields of view at least partially overlap; the image processor configured to receive a first image and a second image from the first imaging device and the second imaging device, respectively; the image processing device comprising at least one processor and at least one memory containing computer program code for one or more programs; The at least one memory and the computer program code include: acquiring the first image and the second image, the first image being a visible light image and the second image being a thermal image, the first image and the second image at least partially depicting a same real world scene; automatically identifying a corresponding set of one or more spatial portions of the second image that represent a corresponding one of a plurality of different predetermined temperature ranges including a first predetermined temperature range representing a human body temperature and a second predetermined temperature range representing temperatures of objects that may be harmful to humans due to excessive heat if in direct contact or in close proximity thereto, by covering temperatures of 44° C. or greater; for each set of one or more spatial portions of the second image, identifying a corresponding set of one or more spatial portions of the first image that illustrates an identical portion of a real-world scene as shown in the corresponding set of the identified one or more spatial portions of the second image; deriving a composite image based on the first image, the composite image having a different emphasis applied to each of the set of one or more spatial portions of the first image by enhancing the identified set of one or more spatial portions of the first image, and providing the composite image to the display for display; and causing the image processing device to execute, using the at least one processor, Exploration camera equipment.

3. the plurality of different predetermined temperature ranges are defined as corresponding predetermined ranges of pixel values; 3. The exploration camera device of claim 1 or claim 2, wherein identifying the corresponding set of one or more spatial portions of the second image comprises identifying pixel locations of the second image having pixel values ​​that are within the corresponding predetermined range of pixel values.

4. The exploration camera device of any one of claims 1 to 3, wherein identifying the set of one or more spatial portions of the first image includes converting pixel positions within the identified set of one or more spatial portions of the second image to corresponding pixel positions of the first image using a predetermined spatial mapping function.

5. 5. The exploration camera device of claim 4, wherein the predetermined spatial mapping function includes mapping pixel locations of the second image to corresponding pixel locations of the first image by translating the pixel locations by a predetermined amount in a predetermined direction.

6. deriving the composite image, generating an enhanced image including respective visual indications at spatial locations that spatially correspond to the identified set of one or more spatial portions of the first image, with other portions of the enhanced image being transparent; overlaying the enhanced image onto the first image; The exploration camera device according to any one of claims 1 to 5, comprising:

7. The exploration camera device of any one of claims 1 to 5, wherein deriving the composite image includes applying different modifications to each of the set of one or more spatial portions of the first image by modifying image content of at least a portion of the first image in a predetermined manner in the identified set of one or more spatial portions of the first image.

8. The modification is replacing the image content in the identified set of one or more spatial portions of the first image with one or more visual representations that are different for each of the set of one or more spatial portions of the first image, respectively; modifying the image content around the identified set of one or more spatial portions of the first image in a predetermined manner that is different for each of the set of one or more spatial portions of the first image; 8. The surveillance camera device of claim 7, comprising one of:

9. The exploration camera device according to any one of claims 1 to 8, wherein the first imaging device and the second imaging device are arranged such that their respective optical axes are parallel to each other at a predetermined distance from each other.

10. An image processing method for an exploration camera device according to any one of claims 1 to 9, acquiring a first image and a second image showing a same real world scene, the first image comprising a visible light image and the second image comprising a thermal image; automatically identifying a corresponding set of one or more spatial portions of the second image that represent a corresponding one of a plurality of different predetermined temperature ranges including a first predetermined temperature range representing a human body temperature and a second predetermined temperature range representing temperatures of objects that may be harmful to humans due to excessive heat if in direct contact or in close proximity thereto, by covering temperatures of 44° C. or greater; for each set of one or more spatial portions of the second image, identifying a corresponding set of one or more spatial portions of the first image that represents the same portion of a real-world scene as shown in the corresponding set of the identified one or more spatial portions of the second image; deriving a composite image based on the first image, the composite image having a different emphasis applied to each of the set of one or more spatial portions of the first image by enhancing the identified set of one or more spatial portions of the first image, and providing the composite image to the display for display; causing the image processing device to execute the steps of: An image processing method comprising:

11. A computer program for image processing in an exploration camera device according to any one of claims 1 to 9, the computer program comprising computer-readable program code configured to cause the method of claim 10 to be performed when the program code is executed by the at least one processor of the image processing device. A computer program for image processing.

Images