Method and system for detecting objects in a maritime or terrestrial scene

Abstract

The invention relates to a method for detecting objects in a scene, the method comprising the following steps: - acquiring a plurality of image sectors of a scene by means of at least one image sensor (1); - preprocessing each of the image sectors in order to correct at least one image irregularity; - constructing image segments of the scene, each image segment being constructed from at least two preprocessed image sectors; - processing the image segments by means of a processing algorithm according to a type of observed scene; - detecting objects in each image segment by means of a neural network; - reconstructing the image of the scene from the image segments.

Description

[0001] DESCRIPTION

[0002] Title: Method and system for detecting objects in a maritime or terrestrial scene.

[0003] Technical field

[0004] The present invention relates to a method for detecting objects in a maritime and / or terrestrial scene. It also relates to a system for detecting objects in a maritime and / or terrestrial scene.

[0005] State of the prior art

[0006] A relatively ignored topic in the object detection literature is the quality of input images. Images coming out of the camera have a 14-bit dynamic range, and infrared has the particularity of having significant intensity changes across the scene but locally quite low contrasts. Passing an image without prior processing through a detection network does not provide the best detection performance, and many known treatments in the literature do not give good results.

[0007] Additionally, infrared cameras are used in a variety of imaging applications to capture infrared images. For example, infrared cameras can be used for face detection, land object detection, and possibly maritime object detection as a complement.

[0008] For many practitioners in the sector, improving data processing upstream of the detection network is not mandatory and the focus is mainly on the detection network. Many publications propose very simple processing. Some publications seek to optimize image processing (for example for color images) to achieve the best possible detection performance.

[0009] Known in particular from the state of the art is document US2014168439A1. This document describes systems and methods providing processing techniques and operating modes for infrared cameras for maritime applications. For example, an infrared camera system for a watercraft is described for monitoring the perimeter of the watercraft to detect a man overboard condition. The infrared images may be processed, for example, based on the man overboard operating mode or other selectable operating modes for the infrared camera system. An alert may be provided and a spotlight and / or a narrow field of view camera may be directed to an area of ​​interest if a man overboard condition exists.

[0010] There are generally a number of drawbacks to approaches to implementing processing for conventional land and sea detections for infrared cameras, such as the detection of small objects which remains difficult.

[0011] The aim of the present invention is to solve at least one of the drawbacks mentioned.

[0012] Statement of the invention

[0013] This objective is achieved with a method for detecting objects in a scene, the method comprising the following steps:

[0014] - acquisition of a plurality of image sectors of a scene by at least one image sensor,

[0015] - pre-processing of each of the image sectors to correct at least one image irregularity,

[0016] - construction of image segments of the scene, each image segment being constructed from at least two pre-processed image sectors,

[0017] - processing of image segments by a processing algorithm based on a type of scene observed,

[0018] - detection of objects in each image segment by a neural network,

[0019] - reconstruction of the image of the scene from said image segments.

[0020] Advantageously, the at least one image sensor may correspond to a rotating image sensor or to a fixed image sensor.

[0021] Preprocessing removes dead pixels from the sector and reduces noise (photonic, electronic, fixed) from each of the image sectors. An "image sector" is a piece of an image, particularly a so-called panoramic image.

[0022] An image segment is an image constructed from at least two image sectors. Preferably, an image segment is constructed from three image sectors. Constructing image segments rather than using image sectors for detection allows the neural network to detect objects that do not fit on a single image sector.

[0023] The purpose of image segment processing is to homogenize the appearance and dynamics of the images seen by the neural network. This facilitates the network's ability to have maximum performance on unseen scenes. This homogeneity therefore allows for better network performance. For the case of object detection in a terrestrial scene, the processing also adds an image representing the difference of the current image compared to the moving average of the previous images. The addition of this image makes it possible to introduce temporal information useful for detection, particularly for small, blurred or low-contrast objects. Part of the processing for object detection in a terrestrial scene is adapted to make a display option of the image from the image sensor.

[0024] The method may also include the following step:

[0025] - tracking the object(s) detected by a tracking algorithm in said image of the reconstructed scene, the detected object(s) being associated with a tracking category.

[0026] Object tracking involves temporally matching object detections based on motion and similarity.

[0027] Tracking the detected object may include matching the detected object of a detection N with the detected object of a previous detection Nl by the tracking algorithm.

[0028] The detected object(s) may be associated with at least one of the following tracking categories:

[0029] - alarm,

[0030] - track. During the detection stage, the neural network proposes a target class (e.g., sailboat, human, etc.) when an object is detected. The detected object is then classified into the proposed target class. The categorization 'alarm' or 'track' occurs during the tracking stage. A detection by the neural network is associated with an alarm as soon as it is not part of the track category. A track is formed when several alarms have been temporally paired, for example, according to criteria of movement, similarity and a detection score. The tracks therefore correspond to tracks confirmed by pairing of the tracking algorithm. The choice of the class of the detected object is refined during tracking. Indeed, when a detected object is associated with the track category, the chosen class may be the one that was most proposed by the neural network for the object considered.Filtering of categorized objects can be configured by the user.

[0031] According to one embodiment, the preprocessing step may also comprise a step of transforming the image according to a geometric transformation.

[0032] Preprocessing allows to reconstruct, by pasting the contents of the sectors, a so-called panoramic image, corresponding to a horizontal surveillance strip knowing that the images are vertically aligned by property of the image sensor(s) in rotation or not. The appearance of the objects being slightly affected by the deformations, the correction of the deformations before the detection allows to optimize said detection. Indeed, a straight line can for example become a curve following a deformation. It is therefore preferable to correct the image before programming an object detection.

[0033] The step of processing image segments, when a terrestrial scene is observed, may include the following steps, for each of the image segments:

[0034] - converting the image segment according to a first, a second and a third conversion parameter, three output image segments being obtained and then averaged together, - determining a difference image from the image segment and a running average of the converted image segments,

[0035] - conversion of the difference image according to a fourth conversion parameter,

[0036] - concatenation of the converted image segment and difference image.

[0037] The processing for detecting objects in a terrestrial scene, here called "terrestrial processing", or for detecting terrestrial targets, gives a better image. This terrestrial processing allows a very fast implementation on a graphics processor. It also makes it possible to enhance all the contrasts of the image segment, while maintaining a certain local hierarchy of intensities (an object that is brighter than its environment will remain so in a certain neighborhood). Conversely, a classic CLAHE allows, depending on the chosen parameters, either to enhance the local contrasts well, or to maintain a certain local hierarchy of intensities, but not both. CLAHE is a known technique used to improve the local contrast of an image. Moreover, to a certain extent, CLAHE enhances certain details uniformly while they were not of the same contrast in the original image.This is much less the case with the terrestrial processing according to the invention. It produces a fairly natural and pleasant image for a human to look at. It also helps the neural network to correctly detect the objects present in each image segment.

[0038] Still in the case of terrestrial targets, providing temporal information to the image provided to the neural network allows for increased performance in distinguishing low-contrast and / or small objects. The image provided to the neural network corresponds to an image segment processed by the processing.

[0039] Regarding concatenation, this step produces a two-channel image. Passing a multi-channel image through the neural network optimizes detection performance.

[0040] The image segment conversion step may include the following steps:

[0041] - determination of three sets of parameters, for each set of parameters:

[0042] - determination of a transformation from the parameter set,

[0043] - application of the transformation on the image segment.

[0044] The step of processing image segments, when a maritime scene is observed, may include the following steps, for each of the image segments:

[0045] - modification of at least one parameter of the image segment,

[0046] - application of a piecewise affine transformation on the modified image segment.

[0047] Changing at least one parameter of the image segment is achieved by removing low-frequency information from the image, which is equivalent to removing a heavily blurred version of the input image. This modification causes a compression of the dynamic range.

[0048] The processing for maritime scene detection, called "maritime processing", or for the detection of maritime targets, uses an adaptive piecewise affine transformation. Since contrasts in the extremes of the original image dynamics are of little use in maritime target detection, this transformation allows the image dynamics to be concentrated on the essential information necessary for detection.

[0049] According to yet another aspect of the invention, there is provided a system for detecting objects in a maritime or terrestrial scene, the system comprising:

[0050] - at least one image sensor arranged to acquire several image sectors of an observed scene,

[0051] - a processing module arranged and / or programmed to apply the method according to the invention.

[0052] According to yet another aspect of the invention, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to the invention. Description of the figures and embodiments

[0053] Other advantages and particularities of the invention will appear on reading the detailed description of implementations and embodiments which are in no way limiting, and the following appended drawings:

[0054] [Fig. 1] is a schematic view of a system according to one embodiment of the invention.

[0055] [Fig. 2a] is a flowchart of the terrestrial treatment according to one embodiment of the invention.

[0056] [Fig. 2b] is a flowchart of a step of the terrestrial treatment according to one embodiment of the invention.

[0057] [Fig. 3a] is a CLAHE terrestrial image.

[0058] [Fig. 3b] is the first channel of a terrestrial image processed by the terrestrial processing according to an embodiment of the invention.

[0059] [Fig. 4a] is a CLAHE maritime image.

[0060] [Fig. 4b] is a maritime image processed by maritime processing according to one embodiment of the invention.

[0061] Since these embodiments are in no way limiting, it is possible in particular to consider variants of the invention comprising only a selection of characteristics described or illustrated subsequently isolated from the other characteristics described or illustrated (even if this selection is isolated within a sentence comprising these other characteristics), if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection comprises at least one preferably functional characteristic without structural details, and / or with only a part of the structural details if this part alone is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

[0062] We will first describe, with reference to Figure 1, a system configured for detecting objects in a maritime and / or terrestrial scene. The system presented implements the method according to the invention. The system comprises at least one rotating image sensor 1 arranged to acquire several infrared image sectors of a scene. In a first embodiment, the system comprises a single image sensor 1. The image sensor 1 is rotatable in order to acquire several sectors of the environment by following a panorama. The rotating image sensor 1 corresponds, in a preferred embodiment, to an infrared rotating image sensor 1. The image sensor can rotate clockwise and vice versa. The image sensor 1 can rotate through a rotation angle of 360 degrees.

[0063] In another embodiment, the system comprises several image sensors 1 positioned around an axis. Each image sensor 1 is then configured to acquire several sectors of the environment over a limited panorama portion. For example, each image sensor 1 can rotate through a rotation angle of 30 degrees depending on the number of image sensors present. The invention can be applied to other moving or rotating systems as shown.

[0064] According to other embodiments, the image sensor may correspond to one or more fixed image sensors. In the case where a single fixed image sensor is used, it is preferably a wide-angle image sensor. A “wide-angle” image sensor is understood to mean an image sensor that acquires images having a wider viewing angle than that of a conventional image sensor.

[0065] The system also includes a processing module (not shown in Figure 1) arranged and / or programmed to:

[0066] - preprocess each of the image sectors to correct at least one image irregularity,

[0067] - construct image segments of the scene, each image segment being constructed from at least two pre-processed image sectors,

[0068] - process the image segments using a processing algorithm based on the type of scene observed,

[0069] - detect objects in each image segment by a neural network,

[0070] - reconstruct the image of the scene from said image segments.

[0071] When the infrared image sensor 1 rotates, it is configured to acquire several images at different rotation angles. These images, taken at different times, correspond to image sectors. Depending on the image sensor model and its configuration, this can range from a few image sectors to hundreds. At the image sensor output, the processing module is configured to preprocess these image sectors. The image sectors are then transformed by the processing module according to a geometric transformation that corrects the deformations of the image sensor. This geometric transformation positions the pixels of the image sector in a spherical frame of reference. Following this geometric transformation, the pixels on the left edge of an image sector correspond to photons seen by the image sensor coming from the same horizontal angle relative to said image sensor. The same applies to the pixels on the right edge.The processing module is configured to ensure that the relative angle of the intermediate pixels with respect to the angle corresponding to the edges of the image sector is linearly dependent on their pixel coordinate in the image. The same phenomenon is forced vertically.

[0072] With mathematical notations, let x and y coordinates be integers, between o and w - 1 and o and h - 1 respectively, corresponding to the coordinates of a pixel in the image of width w and height h, then the horizontal angle 0 verifies at each pixel that:

[0073] Similarly, the vertical angle cp verifies at each pixel that:

[0074] When the image sector preprocessing step is finalized, the processing module is configured to construct image segments. Each image segment is constructed from three image sectors, ideally without overlap except for the last sector if the number of image sectors is not a multiple of three. In certain embodiments, an image segment is constructed from three image sectors with a partial overlap between each image sector. Partial overlap is understood to mean an overlap in which the overlap of the image sectors is less than a total overlap. Generally, the sizes of the image segments may correspond to 2160x1280 (Spynel X), 1536x640 (Spynel S) or 1920x512 (Spynel M).

[0075] In another embodiment, the processing module may be configured to construct, from the acquired image sectors, a panoramic image of the surrounding scene. In this case, the processing module is then configured to cut said panoramic image into image segments according to the invention.

[0076] The processing module is then configured to separately process the image segments via a processing algorithm depending on the type of scene observed by the image sensor. Three types of scenes can be observed: a maritime scene, a terrestrial scene, a maritime and terrestrial scene. The image segments are processed as imaged for the so-called "maritime" and "terrestrial" processing algorithms. The processing module is therefore configured to process the image segments differently depending on the type of scene observed.

[0077] The processing module is configured to use so-called "maritime" processing when a maritime scene or piece of a scene is observed by the image sensor.

[0078] The processing module is configured to use so-called "terrestrial" processing when a terrestrial scene or piece of a scene is observed by the image sensor.

[0079] Figure 2a illustrates a flowchart of the processing applied by the processing module when a terrestrial scene is observed according to the invention.

[0080] Terrestrial processing is broken down into several steps. The image before detection, i.e. the input image segment, is converted 2a using a multi-scale conversion to obtain an eight-bit image. For each of the image segments, the processing module is configured to:

[0081] - converting 2a the image segment according to a first, a second and a third conversion parameter, three output image segments being obtained and then averaged between them,

[0082] - determine 2b an image of the differences from the image segment and a moving average of the converted image segments,

[0083] - convert 2c the difference image according to a fourth conversion parameter, - concatenate 2d the converted image segment and difference image. Step 2a is divided into sub-steps according to figure 2b.

[0084] The processing module is configured to determine three sets of parameters and for each set of parameters:

[0085] - determine a transformation A2 from the parameter set,

[0086] - apply the A3 transformation to the image segment.

[0087] The processing module is then configured to calculate (step A1 of Figure 2b), for an input image, i.e. an image segment, local histograms on three local neighborhood sizes. The three sizes can correspond to the following sizes for example: size 1: 256x256, size 2: 128x64, size 3: 16x16. A local neighborhood size corresponds to a conversion parameter. Here, we therefore have three conversion parameters defined.

[0088] For the two largest local neighborhood sizes, these histograms are used by the processing module to retrieve predetermined quantiles (step A2 of Figure 2b). In the case of the present invention, the processing module is configured to retrieve the minimum quantile, the maximum quantile, and 1% and 99% quantiles. The processing module is then configured to, on each neighborhood, apply a piecewise affine transformation (step A3 of Figure 2b). That is, the processing module is configured to linearly project onto the interval [0, 25.5] the pixels between the minimum value of the image and the 1% quantile of the image. The processing module is also configured to linearly project onto [25.5, 229.5] the values ​​contained between the 1% quantile and the 99% quantile and the remainder onto [229.5, 255].The processing module, in the case of the smallest size, is configured to perform a linear transformation from minimum to maximum to [0, 255].

[0089] A floating-point image, i.e., an image encoded with floating-point numbers, is output for each local neighborhood size. These images are then averaged across the three local neighborhood sizes and converted to eight-bit integers. The processing module is then configured to compute a moving-average difference image (step 2b of Figure 2a) at this position in the image before detection. To compute the difference image, the processing module subtracts the moving average of the input images (i.e., the image segments before detection) from the input image segment (factor 0.95).

[0090] The processing module is then configured to convert (2c in Figure 2a) the difference image into the moving average using a single-scale conversion. That is, the conversion operates in the same way as explained previously but with a single neighborhood size. The processing module is configured to calculate local histograms on a local neighborhood size. Here, size 2 is used. Size 2 corresponds to the fourth conversion parameter, which is, in this embodiment, similar to one of the first three conversion parameters defined above. Once the image segment has been processed in this way, the processing module is configured to round the pixel values ​​to integers. The processing module can also be configured to then multiply the signal by a constant.

[0091] The duo "current image" and "difference image" produced is concatenated (step 2d of Figure 2a) by the processing module and then transmitted to the neural network for detection. The term "current image" refers to the most recent image segment available, i.e. the input image segment for processing. A subtraction of one hundred and twenty-seven is performed before providing the image to the neural network. That is, the processing module is configured to center the signal, here by subtracting one hundred and twenty-seven.

[0092] Referring to Figure 3a, we can see a CLAHE terrestrial image and, in Figure 3b, the first channel of the terrestrial image of Figure 3a processed by the so-called terrestrial processing according to the invention.

[0093] The so-called "maritime" processing is configured to remove the low-frequency information from the image, i.e. from the image segment, by removing a highly blurred version of the input image, the input image having a fourteen-bit encoding. The processing module is configured to then apply a piecewise affine transformation using the quantiles of the image. More specifically, the processing module is configured to, with the following proposed parameters, linearly project onto the interval [0, 25.5] the pixels between the minimum value of the image and the 1% quantile of the image. The processing module is also configured to linearly project onto [25.5, 229.5] the values ​​contained between the 1% quantile and the 99% quantile and the remainder onto [229.5, 255].Once the image segment has been processed in this way, the processing module is configured to round the pixel values ​​to integers and subtract 127 before providing the image to the neural network. This projection step corresponds to the normalization before running a neural network. That is, the processing module is configured to center the signal, here by subtracting 127. The processing module can also be configured to then multiply the signal by a constant.

[0094] Referring to Figure 4a, we can see a CLAHE maritime image and, in Figure 4b, the maritime image of Figure 4a processed by the so-called maritime processing according to the invention.

[0095] In an embodiment where a maritime and terrestrial scene is observed by the image sensor, both maritime processing and terrestrial processing may be used by the processing module 2.

[0096] When the image segments have been processed according to the type of scene observed, the processing module is configured to detect one or more objects in each processed image segment via a neural network.

[0097] Regarding the neural network, a network architecture based on TTFNet (Training-Time-Friendly Network) is used. Any other compatible network type can be used in other embodiments. The neural network is configured to return a table, for example, of the hundred highest scoring detections. This allows for the elimination of detections below a user-defined threshold. The processing module can also be configured to use an additional algorithm to filter out double detections. In a particular embodiment, the additional algorithm is a “non-maximum suppression” (NMS) algorithm. The detection step then makes it possible to detect the objects present or not in each image segment.

[0098] The processing module is then configured to construct a panoramic image of the scene from the image segments processed by the neural network. The processing module is finally configured to track said detected objects of the reconstructed scene via a tracking algorithm in said image, the detected object(s) being associated with a tracking category.

[0099] Detections are matched with previous detections using the tracking algorithm. Detected objects are then classified into one of the following classes: alarm or track. The objects are then filtered according to user-defined rules and transmitted to the user. The panoramic image can be displayed to the user, and the detected objects can be displayed as an overlay, for example, on this panoramic image.

[0100] Typically at least one of the means of the device according to the invention previously described, preferably each of the means of the device according to the invention previously described, is a technical means.

[0101] Typically, each of the means of the device according to the invention previously described may comprise at least one computer, a central or calculation unit, an analog electronic circuit (preferably dedicated), a digital electronic circuit (preferably dedicated) and / or a microprocessor (preferably dedicated) and / or software means.

[0102] Of course, the invention is not limited to the examples which have just been described and numerous adjustments can be made to these examples without departing from the scope of the invention.

[0103] Of course, the various embodiments, features, forms and variants of the invention may be combined with each other in various combinations provided that they are not incompatible or mutually exclusive. In particular, all the embodiments and variants described above may be combined with each other.

Claims

CLAIMS 1. A method of detecting objects in a scene, the method comprising the following steps: - acquisition of a plurality of image sectors of a scene by at least one image sensor (1), - pre-processing of each of the image sectors to correct at least one image irregularity, - construction of image segments of the scene, each image segment being constructed from at least two pre-processed image sectors, - processing of image segments by a processing algorithm based on a type of scene observed, - detection of objects in each image segment by a neural network, - reconstruction of the image of the scene from said image segments.

2. Method according to claim 1, wherein the at least one image sensor corresponds to a rotating image sensor or a fixed image sensor.

3. A method according to any preceding claim, wherein the method also comprises the following step: - tracking the object(s) detected by a tracking algorithm in said image of the reconstructed scene, the detected object(s) being associated with a tracking category.

4. The method of claim 3, wherein tracking the detected object comprises matching the detected object of a detection N with the detected object of a previous detection Nl by the tracking algorithm.

5. Method according to any one of the preceding claims, in which the detected object(s) are associated with at least one of the following tracking categories: - alarm, - track.

6. Method according to any one of the preceding claims, in which the preprocessing step also comprises a step of transforming the image according to a geometric transformation.

7. Method according to any one of the preceding claims, in which the step of processing the image segments, when a terrestrial scene is observed, comprises the following steps, for each of the image segments: - conversion (2a) of the image segment according to a first, a second and a third conversion parameter, three output image segments being obtained and then averaged between them, - determination of a difference image (2b) from the image segment and a moving average of the converted image segments, - conversion (2c) of the difference image according to a fourth conversion parameter, - concatenation (2d) of the converted image segment and difference image.

8. The method of claim 7, wherein the step of converting the image segment comprises the following steps: - determination (Al) of three sets of parameters, For each parameter set: - determination of a transformation (A2) from the set of parameters, - application of the transformation (A3) on the image segment.

9. A method according to any preceding claim, wherein the step of processing the image segments, when a maritime scene is observed, comprises the following steps, for each of the image segments: - modification of at least one parameter of the image segment, - application of a piecewise affine transformation on the modified image segment.

10. System for detecting objects in a maritime or terrestrial scene, the system comprising: - at least one image sensor (1) arranged to acquire several image sectors of an observed scene, - a processing module arranged and / or programmed to apply the method according to any one of claims 1 to 9.

11. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to any one of claims 1 to 9.

Images