Object detection using multiple imagers with distinct spectral bands

Abstract

Detection method, using an optronic device comprising several imagers (4) operating in distinct frequency bands, comprising the steps of: - acquiring at least two primary images (Ip) of the same scene produced by at least two different imagers; - registering the at least two primary images, to obtain, for each primary image, a secondary image; - producing a multispectral image (Im) from the secondary images, and detecting a target (8) in the multispectral image; - producing a detection image (Id) in which a visual indicator is displayed. FIGURE OF THE ABSTRACT: Fig.3

Description

Title of the invention: Object detection using several imagers with distinct spectral bands

[0001] The invention relates to the field of optronic devices integrating imagers and used for object detection operations (target decamouflage for example).

[0002] BACKGROUND OF THE INVENTION

[0003] Optronic devices are known which comprise a plurality of cameras operating on distinct frequency bands. These cameras comprise, for example, one or more of the following cameras: thermal camera, color camera, NIR camera (for Near Infrared, i.e. near infrared), SWIR camera (for Short-Wave Infrared, i.e. short infrared), etc.

[0004] Such optronic devices are also known which are further equipped with a spectro-imager, which may be, for example, a multispectral (or super-spectral or hyper-spectral) camera. The spectro-imager is intended, for example, to carry out target decamouflage operations (such as an armored vehicle, for example).

[0005] Multispectral imaging, for example, thus consists of discretely acquiring the energy reflected or emitted by a surface in a number of spectral bands typically between 3 and 20. The spectral bands can be contiguous (but not necessarily).

[0006] The acquisition is carried out by a multispectral camera comprising a multispectral sensor capable of measuring luminance spectra, or even estimating reflectance, in wavelength ranges corresponding to spectral bands located for example in the visible range and / or in the infrared range.

[0007] Document FR 2 982 393 A1 thus describes a method aimed at searching for a target in a multispectral image.

[0008] Known decamouflage methods using multispectral imaging have certain drawbacks.

[0009] As we have just seen, these methods require the use of a multispectral camera, which must therefore be integrated into the optronic device, which increases the size, cost and complexity of the latter.

[0010] Furthermore, in such a device, the result of the decamouflage is embedded in a multispectral image, said multispectral image then being returned to the operator. This therefore requires an operator, who observes a scene via a “conventional” channel » (thermal, color, etc.), to be passed on a channel of the multispectral camera to see the result of the decamouflage.

[0011] Furthermore, the result of the detection, embedded in the multispectral image, is not very intuitive for the operator to interpret.

[0012] The multispectral camera also needs to be calibrated very finely to guarantee good target detection results.

[0013] SUBJECT OF THE INVENTION

[0014] The subject of the invention is a detection method which is effective and which provides a result which is easy to interpret, and the implementation of which uses a simple, inexpensive, small-volume optronic device, and which does not require complex calibration operations other than the usual calibration operations of the channels used. Summary of the invention

[0015] With a view to achieving this aim, a detection method is proposed, using an optronic device comprising several imagers operating in distinct frequency bands, the detection method being implemented in a processing unit and comprising the steps of:

[0016] - acquire at least two primary images of the same scene produced by at least two different imagers;

[0017] - realign the at least two primary images, to obtain, for each image primary, a secondary image derived from said primary image;

[0018] - produce a multispectral image from the secondary images;

[0019] - detect a target in the multispectral image;

[0020] - produce a detection image, formed from one of the primary images or of one of the secondary images, in which a visual indicator is displayed, a position of the visual indicator in the detection image corresponding to a position of the target in the scene.

[0021] The detection method therefore does not require the use of an optronic device integrating a multispectral imager. The multispectral image is in fact produced from the secondary images, which are obtained by recalibrating the primary images, themselves produced by imagers each capable of operating on a single spectral band.

[0022] The optronic device used to implement the detection method is therefore simple, inexpensive and compact, since it does not incorporate a multispectral imager. The imagers of the optronic device do not require complex calibration operations other than the usual calibration operations of the channels used.

[0023] The detection image is formed from an image obtained via a conventional channel. The operator can therefore have access to the result of the detection while continuing to observe the scene via this conventional channel. The result is also easier for the operator to interpret.

[0024] We further propose a detection method as previously described, the step of registering the at least two primary images comprising the steps of:

[0025] - measure a detection distance between the optronic device and a point on the scene targeted by the optronic device;

[0026] - access a recalibration table comprising recalibration coefficients associated with reference distances;

[0027] - realign the primary images to obtain the secondary images using the associated recalibration coefficients in the recalibration table at a reference distance which is closest to the detection distance.

[0028] Further provided is a detection method as previously described, wherein the visual indicator is displayed in a primary image to produce the detection image, the detection method comprising the steps of:

[0029] - create a first intermediate image in a reference frame of the secondary image, the first intermediate image including the visual indicator;

[0030] - use the registration table to create, from the first intermediate image, a second intermediate image in a reference frame of the primary image;

[0031] - integrate into the primary image the visual indicator included in the second image intermediate.

[0032] We further propose a detection method as previously described, the step of registering the at least two primary images comprising the steps of:

[0033] - detecting in the primary images markers common to said images primary;

[0034] - calculate geometric transformations, each associated with a primary image, which allow common benchmarks to be superimposed;

[0035] - apply geometric transformations to the primary images to obtain the secondary images.

[0036] Further provided is a detection method as previously described, wherein the visual indicator is displayed in a primary image to produce the detection image, the detection method comprising the steps of:

[0037] - calculate an inverse geometric transformation of the geometric transformation associated with said primary image;

[0038] - apply said inverse geometric transformation to the target to obtain the position of the visual indicator in said primary image.

[0039] A detection method as previously described is further provided, wherein the visual indicator is displayed in a secondary image to produce the detection image, the position of the visual indicator corresponding to the position of the target in the multispectral image.

[0040] A detection method as previously described is further provided, in which the visual indicator is the target itself.

[0041] A detection method as previously described is further provided, in which the visual indicator is displayed transparently in the primary image or the secondary image from which the detection image is formed.

[0042] We further propose a detection method as previously described, further comprising the steps of:

[0043] - define an analysis zone in the multispectral image;

[0044] - maximize a spectral contrast in the analysis area,

[0045] the target being detected in said analysis zone.

[0046] We further propose an optronic device, comprising:

[0047] - several imagers each operating in at least one frequency band distinct;

[0048] - a processing unit in which the detection method is implemented such as previously described.

[0049] A computer program is further provided comprising instructions which cause the processing unit of the optronic device as previously described to execute the steps of the detection method as previously described.

[0050] A computer-readable recording medium is further provided, on which the computer program as previously described is recorded.

[0051] The invention will be better understood in light of the following description of a particular non-limiting embodiment of the invention. Brief description of the drawings

[0052] Reference will be made to the attached drawings, among which:

[0053] [Fig-1] [Fig.l] represents an optronic device and a screen;

[0054] [Fig.2] [Fig.2] represents steps of the detection method;

[0055] [Fig.3] [Fig.3] illustrates some of said steps;

[0056] [Fig.4] [Fig.4] represents examples of primary images and images secondary;

[0057] [Fig.5] [Fig.5] illustrates the operational implementation of the detection method. DETAILED DESCRIPTION OF THE INVENTION

[0058] With reference to [Fig.l], the detection method is implemented in an optronic device 1 which is connected to a screen 2 (the screen 2 could be integrated into the optronic device 1). The optronic device 1 and the screen 2 are here mounted on a land vehicle 3. The detection method here is a decamouflage method aimed at detecting targets, such as armored vehicles, weapons, etc.

[0059] The optronic device 1 comprises a plurality of cameras 4 and a processing unit 5.

[0060] The cameras 4 operate in separate frequency bands.

[0061] This means that:

[0062] - each camera 4 operates in one or more frequency bands, and

[0063] - this or these frequency bands comprise at least one sub-band of frequencies that are not included in any of the frequency bands in which the other cameras operate 4.

[0064] The cameras 4 comprise N cameras, which therefore operate in at least N distinct frequency bands.

[0065] Here, N = 4.

[0066] The cameras 4 include a camera 4a operating in the visible spectrum (VIS), a camera 4b operating in the near infrared spectrum (NIR), a camera 4c operating in the short infrared spectrum (SWIR), and a thermal camera 4d.

[0067] The processing unit 5 comprises at least one processing component 5a, which is for example a “generalist” processor, a processor specialized in signal processing (or DSP, for Digital Signal Processing), a microcontroller, or a programmable logic circuit such as an FPGA (for Field Programmable Gate Arrays) or an ASIC (for Application Specific Integrated Circuit).

[0068] The processing unit 5 further comprises one or more memories 5b, connected to or integrated in the processing component 5a. At least one of these memories 5b forms a computer-readable recording medium, on which is recorded at least one computer program comprising instructions which cause the processing component 5a to execute at least some of the steps of the detection method which will be described.

[0069] With reference to figures 2 and 3, the processing unit 5 acquires at least two primary images Ip of the same scene produced by at least two different cameras 4: step EL

[0070] Here, the processing unit 5 acquires, for each camera 4, a primary image Ip produced by said camera 4.

[0071] The processing unit 5 therefore acquires N primary images Ip, i.e. four primary images Ip.

[0072] The acquisition of the primary images Ip is carried out synchronously. It is the processing unit 5 which manages this synchronization here.

[0073] The way in which the synchronization is implemented may depend on the support on which the optronic device 1 may be mounted. This is the land vehicle 3. Thus, if the support is fixed or is moving at low speed, the primary images Ip produced by the different cameras 4 may be acquired successively in a few seconds.

[0074] It is noted that the cameras 4 used do not necessarily need to have a similar optical field, but the detection method is applied in the areas of field overlap between the cameras 4. Indeed, the areas analyzed must be present in the primary images Ip of the cameras 4 used.

[0075] As can be seen in [Fig.3], the target (here a battle tank) is possibly difficult or impossible to distinguish on the primary images Ip, i.e. via conventional channels.

[0076] Then, the processing unit 5 realigns the primary images Ip to obtain, for each primary image Ip, a secondary image Is from said primary image Ip: step E2.

[0077] It is noted that it is possible, for a primary image Ip (or several), that the secondary image Is resulting from said primary image is said primary image itself. This situation may occur for example in the case of a projection of one or more primary images onto another primary image, without modifying said other primary image.

[0078] Registration consists of spatially matching different images of the same scene, produced by the different imagers. We thus obtain, from the primary images Ip, secondary images Is defined in relation to the same reference frame.

[0079] The registration is a digital registration, which can be carried out in different ways.

[0080] The processing unit 5 can measure a detection distance between the optronic device 1 and a point of the scene targeted by the optronic device 1.

[0081] The processing unit 5 then accesses a recalibration table 7 comprising recalibration coefficients associated with reference distances.

[0082] The resetting table 7 is here stored in one of the memories 5b of the processing unit 5.

[0083] The registration coefficients make it possible to define, for each pixel of each secondary image Is, the coordinates of the pixel of the primary image Ip to be associated with it.

[0084] If these coordinates are whole numbers, it is sufficient to take the value of the pixel corresponding to these coordinates in the image Ip.

[0085] If these coordinates are floating numbers, it is then necessary to interpolate the value of the pixels neighboring these coordinates to obtain the value to be reported in the image Is.

[0086] The registration coefficients mentioned above can be homotheties, resizing factors, elastic transformation factors, etc. They can be variable from one pixel to another of the pixel matrix (in particular in the case of optical distortion).

[0087] The geometry of the system being assumed to be rigid (distance and potential inclination invariable between the sensors), these parameters can be defined once and for all at a given distance, in the factory for example.

[0088] The processing unit 5 then recalibrates the primary images Ip using the recalibration coefficients associated in the recalibration table 7 with the reference distance which is closest to the detection distance.

[0089] Alternatively, the processing unit 5 detects in the primary images Ip reference points common to said primary images Ip. The processing unit 5 then calculates geometric transformations, each associated with a primary image Ip, which make it possible to superimpose the common reference points.

[0090] The processing unit 5 then applies the geometric transformations to the primary images Ip to obtain the secondary images Is.

[0091] In [Fig.4] we see examples of primary images Ip: SWIR image, NIR image, image in the green frequency band, thermal image. We see the associated secondary images Is.

[0092] Then, the processing unit 5 produces a multispectral image Im (hypercube) from the secondary images Is: step E3.

[0093] The hypercube Im is a mathematical object of dimension n. In our case, it is an object of dimension 3.

[0094] The first two dimensions correspond to the spatial resolution of the image, while the third dimension corresponds to the spectral dimension.

[0095] Thus, at each point of the image (defined by the first two dimensions), there corresponds a spectrum as a function of the wavelength.

[0096] The hypercube is therefore a superposition of “spectral planes” of the same scene, each spectral plane being associated with a distinct wavelength.

[0097] The processing unit 5 then samples an analysis zone in the multispectral image Im: step E4. The analysis zone is again a three-dimensional object: two spatial dimensions and a third dimension corresponding to the spectral planes. The analysis zone is therefore a hypercube corresponding to a part of the scene, comprising the same spectral planes as the multispectral image Im.

[0098] The analysis area can be defined in multiple ways. The processing unit 5 can, for example, scan the entire multispectral image and produce, for each point of the image, an analysis area. The processing unit 5 can also select a particular area pointed to, for example, by the operator.

[0099] The processing unit 5 then performs an operation consisting of maximizing the spectral contrast in the analysis zone: step E5.

[0100] This operation is for example carried out according to the method described in document FR 2 982 393 A1.

[0101] For each point of the analysis zone, the processing unit 5 defines for example a target zone around said point. The contrast is calculated for this point between points which belong to this target zone and points of a background zone which is separated from the target zone, inside the analysis zone. The background zone can be separated from the target zone inside the analysis zone by an intermediate separation zone which surrounds the target zone.

[0102] The processing unit 5 carries out, for example, a projection of the vector of spectral intensities for each point of the analysis zone, onto an optimal projection direction in a multidimensional space of these spectral intensities (Fisher projection).

[0103] The processing unit 5 then detects a target 8 in one of the analysis zones of the multispectral image Im.

[0104] By “detect” is meant here: highlighting an area of an image presenting any interest for the observer, for example a spectral anomaly.

[0105] The processing unit 5 then produces a detection image Id, formed from one of the primary images Ip or one of the secondary images Is, in which a visual indicator 9 is displayed: step E6. The position of the visual indicator 9 in the detection image Id corresponds to a position of the target 8 in the scene.

[0106] The visual indicator 9 may be the target 8 itself (more precisely, an image of the target extracted from the multispectral image). This is the case in [Fig.3].

[0107] The visual indicator 9 may also be a simple indicator such as a cross or any other predefined shape (star, rectangle, etc.). It may also be the outline of the target 8. The visual indicator 9 may also be a predetermined shape or object depending on characteristics of the target (size, type, function, number, etc.).

[0108] The visual indicator 9 is embedded in the primary image Ip or the secondary image Is from which the detection image Id is formed.

[0109] The visual indicator 9 can also be displayed transparently in the primary image Ip or the secondary image Is from which the detection image Id is formed (i.e. the part of the scene included within the visual indicator 9 in the detection image Id remains apparent on said image).

[0110] The detection image Id can therefore be obtained from a primary image Ip. The decamouflage is then displayed in said primary image Ip, that is to say in an image which has not undergone the registration.

[0111] This solution makes it possible not to modify the visual appearance of conventional tracks to which the operator is accustomed.

[0112] We consider the case where the registration was carried out using the registration table 7. As we have seen, the registration table 7 is a table of correspondence of the coordinates of the two spaces (space of the primary image Ip and space of the secondary image Is).

[0113] The processing unit 5 therefore has the capacity to find, for each pixel of the primary image Ip, the coordinates of the corresponding image Is (potentially involving an interpolation) - and vice versa.

[0114] The processing unit 5 therefore creates a first intermediate image Id_l in the reference frame of the secondary image Is. The first intermediate image Id_l comprises the visual indicator 9. Then, the processing unit 5 uses the registration table 7, and therefore the correspondence of the coordinates, to create from the first intermediate image Id_l a second intermediate image Id_2 in the reference frame of the primary image Ip.

[0115] The processing unit 5 then integrates the visual indicator 9, included in the second intermediate image Id_2, into the primary image Ip (which are both in the same reference frame), to produce the detection image Id.

[0116] If the resetting has been carried out using geometric transformations, the processing unit 5 calculates a geometric transformation inverse to the geometric transformation associated with said primary image Ip. The processing unit 5 then applies said inverse geometric transformation to the target 8 to obtain the position of the visual indicator 9 in said primary image Ip.

[0117] The detection image Id can also be obtained from a secondary image Is. The decamouflage is therefore displayed in said secondary image Is, that is to say in an image which has undergone the registration.

[0118] The position of the visual indicator 9 then corresponds to the position of the target 8 in the multispectral image Im.

[0119] This solution allows the operator to view the target 8 in the scene in a space where all the channels of the optronic device 1 are aligned, and therefore to move from one to the other without losing reference points in the scene.

[0120] It is noted that it is possible that the registration treatments degrade the definition or the colorimetry of the secondary image Is. The incrustation in the primary image Ip makes it possible to present a detection image to the operator without degradation.

[0121] The processing unit 5 then returns the detection image Id to the operator: step E7.

[0122] With reference to [Fig.5], we are now interested in the operational implementation of the detection method.

[0123] The operator observes a scene, for example in a color imaging channel. He aims at a point of the scene with a reticle 10 of the optronic device 1.

[0124] The operator presses a button which triggers the decamouflage calculation.

[0125] The processing unit 5 then acquires the primary images Ip (comprising the image current primary image Ipc of the color imaging channel that the operator is currently using), and realigns the primary images Ip to obtain the secondary images Is. The processing unit 5 produces the multispectral image Im, and performs the decamouflage.

[0126] The processing unit 5 displays the visual indicator 9 in the current primary image Ipc to produce the detection image Id. The visual indicator 9 here comprises an image of the target 8 itself, as well as a rectangle surrounding the target and comprising a message intended for the operator (for example “Anomaly detection confirmed”). The detection image Id is therefore an “augmented” primary image Ip on which the target 8 is represented, in its true position in the scene, with possibly one or more additional information.

[0127] It is noted that the invention is very advantageous in the case where it is desired to design an optronic device capable of providing images in an imposed spectral band, and of adding a decamouflage function to it, and this, in a restricted volume.

[0128] The predefined images are for example a color image and a thermal image. The restricted volume corresponds for example to the volume of two cameras.

[0129] Two solutions are then possible.

[0130] A first solution would consist of reconstructing a color image from a multispectral camera which would allow the decamouflage to be carried out. However, a degraded spatial resolution and sensitivity of the color image are obtained.

[0131] The second solution, which is the one described here, consists of using color and thermal cameras to perform the decamouflage. The color image is obtained from the color camera, without degradation of the spatial resolution or the sensitivity.

[0132] The detection method and the optronic device can be used in many applications, and in particular for applications in the field of defense: land defense, aeronautical defense, naval defense. The method can be used for target detection, decamouflage and fusion of information from several channels.

[0133] The detection method and the optronic device can also be used for civilian applications.

[0134] These civil applications include, for example, personal rescue: man overboard detection, victim detection in a natural environment. The optronic device is then, for example, mounted on a helicopter or a drone.

[0135] These civil applications also include, for example, biomedical applications, camera detection applications for vehicle driving assistance (ADAS), and space applications (satellite detection, for example).

[0136] These civil applications also include, for example, applications in the field of agronomy, and for example, the detection of anomalies in agricultural territory. The optronic device is then, for example, mounted on a drone to detect obstacles, stones, vegetation, etc.

[0137] Of course, the invention is not limited to the embodiment described but encompasses any variant falling within the scope of the invention as defined by the claims.

[0138] The optronic device is not necessarily mounted on a land vehicle. It can be mounted on any type of vehicle, and even on any type of support (not necessarily mobile).

[0139] The cameras may operate in frequency bands other than those discussed herein. One or more cameras could operate in multiple frequency bands (e.g., color camera, RGB-NIR camera).

[0140] Imagers are not necessarily cameras.

[0141] The processing unit in which the detection method is implemented is not necessarily integrated into the optronic device. It could be remote and acquire the primary images via any means of communication.

[0142] The resetting table is not necessarily stored in a memory of the processing unit. The processing unit could access it by any means of communication.

[0143] The detected “target” can be any object, of any type.

Claims

Claims

1. A detection method, using an optronic device (1) comprising several imagers (4) operating in distinct frequency bands, the detection method being implemented in a processing unit (5) and comprising the steps of: - acquiring at least two primary images (Ip) of the same scene produced by at least two different imagers; - registering the at least two primary images, to obtain, for each primary image, a secondary image (Is) from said primary image; - producing a multispectral image (Im) from the secondary images (Is); - detecting a target (8) in the multispectral image; - producing a detection image (Id), formed from one of the primary images or one of the secondary images, in which a visual indicator (9) is displayed, the position of the visual indicator (9) in the detection image corresponding to a position of the target (8) in the scene.

2. Detection method according to claim 1, the step of registering the at least two primary images comprising the steps of: - measuring a detection distance between the optronic device (1) and a point of the scene targeted by the optronic device; - accessing a registration table (7) comprising registration coefficients associated with reference distances; - registering the primary images to obtain the secondary images (Is) using the registration coefficients associated in the registration table (7) with a reference distance which is closest to the detection distance.

3. A detection method according to claim 2, wherein the visual indicator (9) is displayed in a primary image (Ip) to produce the detection image (Id), the detection method comprising the steps of: - creating a first intermediate image in a frame of reference of the secondary image (Is), the first intermediate image comprising the visual indicator (9); - use the registration table (7) to create, from the first intermediate image, a second intermediate image in a reference frame of the primary image (Ip); - integrate into the primary image (Ip) the visual indicator included in the second intermediate image.

4. Detection method according to claim 1, the step of registering the at least two primary images comprising the steps of: - detecting in the primary images (Ip) reference points common to said primary images; - calculating geometric transformations, each associated with a primary image (Ip), which make it possible to superimpose the common reference points; - applying the geometric transformations to the primary images to obtain the secondary images (Is).

5. A detection method according to claim 4, wherein the visual indicator (9) is displayed in a primary image (Ip) to produce the detection image (Id), the detection method comprising the steps of: - calculating an inverse geometric transformation of the geometric transformation associated with said primary image (Ip); - applying said inverse geometric transformation to the target (8) to obtain the position of the visual indicator (9) in said primary image.

6. Detection method according to one of the preceding claims, wherein the visual indicator (9) is displayed in a secondary image (Is) to produce the detection image (Id), the position of the visual indicator (9) corresponding to the position of the target (8) in the multispectral image (Im).

7. A detection method according to one of the preceding claims, wherein the visual indicator (9) is the target (8) itself.

8. Detection method according to one of the preceding claims, wherein the visual indicator is displayed transparently in the primary image or the secondary image from which the detection image (Id) is formed.

9. Detection method according to one of the preceding claims, further comprising the steps of: - defining an analysis zone in the multispectral image (Im); - maximizing a spectral contrast in the analysis zone, the target (9) being detected in said analysis zone.

10. Optronic device (1), comprising: - several imagers (4) each operating in at least one distinct frequency band; - a processing unit (5) in which the detection method according to one of the preceding claims is implemented.

11. Computer program comprising instructions which cause the processing unit (5) of the optronic device (1) according to claim 10 to execute the steps of the detection method according to one of claims 1 to 9.

12. A computer-readable recording medium on which the computer program according to claim 11 is recorded.

Images