Method and system for detecting objects in a maritime or terrestrial scene.

Abstract

A method for detecting objects in a scene, the method comprising the following steps: - acquisition of a plurality of image sectors of a scene by at least one image sensor (1), - preprocessing of each of the image sectors to correct at least one image irregularity, - construction of image segments of the scene, each image segment being constructed from at least two preprocessed image sectors, - processing of the image segments by a processing algorithm according to a type of observed scene, - detection of objects in each image segment by a neural network, - reconstruction of the image of the scene from said image segments. Figure for the abstract: Fig. 1

Description

Title of the invention: Method and system for detecting objects in a maritime or terrestrial scene. technical field

[0001] The present invention relates to a method for detecting objects in a maritime and / or terrestrial scene. It also relates to a system for detecting objects in a maritime and / or terrestrial scene. Prior art

[0002] A relatively neglected topic in the object detection literature is the quality of input images. Images output from the camera have a 14-bit dynamic range, and infrared is characterized by significant intensity changes across the scene but relatively low local contrast. Passing an image through a detection network without prior processing does not allow for optimal detection performance, and many processing methods described in the literature do not yield good results.

[0003] Furthermore, infrared cameras are used in a variety of imaging applications to capture infrared images. For example, infrared cameras can be used for face detection, terrestrial object detection, and possibly marine object detection as well.

[0004] For many practitioners in the field, improving data processing upstream of the detection network is not mandatory, and attention is primarily focused on the detection network itself. Numerous publications propose very simple processing methods. Some publications aim to optimize image processing (for example, for color images) to achieve the best possible detection performance.

[0005] Prior art is known in particular from the US2014168439A1 document. This document describes systems and methods providing processing techniques and operating modes for infrared cameras for marine applications. For example, an infrared camera system for a vessel is described for monitoring the perimeter of the vessel to detect a man overboard situation. The infrared images can be processed, for example, based on the man overboard operating mode or other selectable operating modes for the infrared camera system. An alert can be provided, and a spotlight and / or a narrow-field camera can be directed toward an area of ​​interest if a man overboard situation exists.

[0006] There are generally a number of disadvantages to the approaches to implementation of processing intended for conventional land and sea detection for infrared cameras, such as the detection of small objects which remains difficult.

[0007] The object of the present invention is to resolve at least one of the aforementioned drawbacks. Description of the invention

[0008] This objective is achieved with a method for detecting objects in a scene, the method comprising the following steps:

[0009] - acquisition of a plurality of image sectors of a scene by at least one sensor of images,

[0010] - preprocessing of each of the image sectors to correct at least one irregularity image regularity,

[0011] - construction of image segments of the scene, each image segment being constructed from at least two pre-processed image sectors,

[0012] - processing of image segments by a processing algorithm based on a type of scene observed,

[0013] - object detection in each image segment by a neural network,

[0014] - reconstruction of the scene image from said image segments.

[0015] Advantageously, at least one image sensor can correspond to a rotating image sensor or a fixed image sensor.

[0016] Preprocessing makes it possible to remove dead pixels from the sector and to reduce noise (photonic, electronic, fixed) from each of the image sectors.

[0017] The term "image sector" means a piece of an image, and in particular of a so-called panoramic image.

[0018] An "image segment" is defined as an image constructed from at least two image sectors. Preferably, an image segment is constructed from three image sectors. Constructing image segments rather than using image sectors for detection allows the neural network to detect objects that do not fit within a single image sector.

[0019] Image segment processing aims to homogenize the appearance and dynamics of the images seen by the neural network. This facilitates the network's ability to achieve maximum performance on unseen scenes. This homogeneity thus allows for better network performance. In the case of object detection in a terrestrial scene, the processing also adds an image representing the difference between the current image and the moving average of previous images. The addition of this image introduces temporal information useful for detection, particularly for small, blurry, or low-contrast objects. Part of the processing for object detection in a terrestrial scene is suitable for making an image display option for the image sensor.

[0020] The process may also include the following step:

[0021] - tracking of the object(s) detected by a tracking algorithm in said image of the scene reconstructed, the detected object(s) being associated with a tracking category.

[0022] Object tracking consists of temporally matching object detections based on movement and similarity.

[0023] The tracking of the detected object may include matching the detected object of a detection N with the detected object of a previous detection Nl by the tracking algorithm.

[0024] The detected object(s) can be associated with at least one of the following tracking categories:

[0025] - alarm,

[0026] - track.

[0027] During the detection stage, the neural network proposes a target class (e.g., sailboat, human, etc.) when an object is detected. The detected object is then classified into the proposed target class. The categorization as 'alarm' or 'track' occurs during the tracking stage. A detection by the neural network is associated with an alarm if it does not belong to the track category. A track is formed when several alarms have been temporally paired, for example, according to criteria such as movement, similarity, and a detection score. Tracks therefore correspond to tracks confirmed by pairing in the tracking algorithm. The choice of the detected object's class is refined during tracking. Indeed, when a detected object is associated with the track category, the chosen class may be the one most frequently proposed by the neural network for the object in question.Users can configure filtering of categorized objects.

[0028] According to one embodiment, the preprocessing step may also include a step of transforming the image according to a geometric transformation.

[0029] Preprocessing allows the reconstruction, by stitching the contents of the sectors, of a so-called panoramic image, corresponding to a horizontal monitoring strip, given that the images are vertically aligned by a property of the image sensor(s), whether rotating or not. Since the appearance of objects is slightly affected by distortions, correcting these distortions before detection optimizes said detection. Indeed, a straight line can, for example, become a curve following a distortion. It is therefore preferable to correct the image before programming object detection.

[0030] The image segment processing step, when a terrestrial scene is observed, may include the following steps, for each of the image segments:

[0031] - conversion of the image segment according to a first, a second and a third conversion parameter, three output image segments are obtained and then averaged together,

[0032] - determination of a difference image from the image segment and a moving average of converted image segments,

[0033] - conversion of the image of differences according to a fourth conversion parameter,

[0034] - concatenation of the image segment and the converted difference image.

[0035] The processing for object detection in a terrestrial scene, here called "Ground processing," or ground-based target detection, produces a better image. This ground processing allows for very fast implementation on a graphics processing unit (GPU). It also enhances all contrasts within the image segment, while maintaining a certain local intensity hierarchy (an object brighter than its surroundings will remain brighter within a certain neighborhood). Conversely, conventional CLAHE, depending on the chosen parameters, can either significantly enhance local contrasts or maintain a certain local intensity hierarchy, but not both. CLAHE is a well-known technique used to improve the local contrast of an image. Furthermore, to a certain extent, CLAHE enhances certain details uniformly, whereas they did not have the same contrast in the original image. This is much less the case with the ground processing according to the invention.It produces a fairly natural and pleasing image for a human to look at. It also helps the neural network to correctly detect the objects present in each image segment.

[0036] Still in the case of terrestrial targets, adding temporal information to the image provided to the neural network allows for increased performance in distinguishing low-contrast and / or small objects. The image provided to the neural network corresponds to an image segment processed by the processing.

[0037] With regard to concatenation, this step produces a two-channel image. Passing a multi-channel image through the neural network optimizes detection performance.

[0038] The image segment conversion step may include the following steps:

[0039] - determining three sets of parameters,

[0040] for each set of parameters:

[0041] - determination of a transformation from the parameter set,

[0042] - application of the transformation on the image segment.

[0043] The image segment processing step, when a maritime scene is observed, may include the following steps, for each of the image segments:

[0044] - modification of at least one parameter of the image segment,

[0045] - application of a piecewise affine transformation on the image segment amended.

[0046] Modifying at least one parameter of the image segment is achieved by removing low-frequency information from the image, which is equivalent to removing a heavily blurred version of the input image. This modification causes a compression of the dynamic range.

[0047] The processing for maritime scene detection, called "maritime processing," or for the detection of maritime targets, uses an adaptive piecewise affine transformation. Since contrasts at the extremes of the original image's dynamic range are not very necessary for maritime target detection, this transformation allows the image's dynamic range to be concentrated on the essential information required for detection.

[0048] According to yet another aspect of the invention, a system for detecting objects in a maritime or terrestrial scene is proposed, the system comprising:

[0049] - at least one image sensor arranged to acquire several image sectors of an observed scene,

[0050] - a processing module arranged and / or programmed to apply the process according to the invention.

[0051] According to yet another aspect of the invention, a computer program product is proposed comprising instructions which, when the program is executed by a computer, lead the latter to implement the steps of the process according to the invention. Description of the figures and methods of realization

[0052] Other advantages and features of the invention will become apparent upon reading the detailed description of implementations and embodiments, which are by no means limiting, and the following accompanying drawings:

[0053] [Fig-1] is a schematic view of a system according to one embodiment of the invention.

[0054] [Fig.2a] is a flowchart of the terrestrial treatment according to an embodiment of the invention.

[0055] [Fig.2b] is a flowchart of a step of the terrestrial treatment according to an embodiment of the invention.

[0056] [Fig.3a] is a CLAHE terrestrial image.

[0057] [Fig.3b] is the first channel of a terrestrial image processed by terrestrial processing according to an embodiment of the invention.

[0058] [Fig.4a] is a CLAHE maritime image.

[0059] [Fig.4b] is a maritime image processed by maritime processing according to a mode of realization of the invention.

[0060] These embodiments are in no way limiting; in particular, variants of the invention comprising only a selection of characteristics may be considered. Features described or illustrated hereafter, isolated from other described or illustrated features (even if this selection is isolated within a sentence including these other features), if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the prior art. This selection includes at least one preferably functional feature without structural details, and / or with only a portion of the structural details if this portion alone is sufficient to confer a technical advantage or to differentiate the invention from the prior art.

[0061] We will first describe, with reference to [Fig. 1], a system configured for detecting objects in a marine and / or terrestrial scene. The presented system implements the method according to the invention. The system comprises at least one rotating image sensor 1 arranged to acquire several infrared image sectors of a scene. In a first embodiment, the system comprises a single image sensor 1. The image sensor 1 is rotatable in order to acquire several sectors of the environment by following a panorama. In a preferred embodiment, the rotating image sensor 1 corresponds to a rotating infrared image sensor 1. The image sensor can rotate clockwise and counterclockwise. The image sensor 1 can rotate through an angle of 360 degrees.

[0062] In another embodiment, the system comprises several image sensors 1 positioned around an axis. Each image sensor 1 is then configured to acquire several sectors of the environment over a limited panoramic portion. For example, each image sensor 1 can rotate through an angle of 30 degrees depending on the number of image sensors present. The invention can be applied to other systems moving or rotating as described.

[0063] According to other embodiments, the image sensor may consist of one or more fixed image sensors. If only one fixed image sensor is used, it is preferably a wide-angle image sensor. A "wide-angle" image sensor is defined as one that acquires images with a wider angle of view than a conventional image sensor.

[0064] The system also includes a processing module (not shown in [Fig. 1]) arranged and / or programmed to:

[0065] - preprocess each of the image sectors to correct at least one irregularity image,

[0066] - construct image segments of the scene, each image segment being constructed from at least two pre-processed image sectors,

[0067] - to process image segments using a processing algorithm based on a type of the observed scene

[0068] - detecting objects in each image segment using a neural network,

[0069] - reconstruct the image of the scene from said image segments.

[0070] During the rotation of the infrared image sensor 1, it is configured to acquire several images at different rotation angles. These images, taken at different times, correspond to image sectors. Depending on the image sensor model and its settings, this can range from a few image sectors to hundreds. At the image sensor output, the processing module is configured to preprocess these image sectors. The image sectors are then transformed by the processing module using a geometric transformation that corrects image sensor distortions. This geometric transformation positions the pixels of the image sector in a spherical coordinate system. Following this geometric transformation, the pixels on the left edge of an image sector correspond to photons seen by the image sensor originating from the same horizontal angle relative to said image sensor. The same applies to the pixels on the right edge.The processing module is configured so that the relative angle of intermediate pixels with respect to the angle corresponding to the edges of the image sector is linearly dependent on their pixel coordinate in the image. The same phenomenon is enforced vertically.

[0071] With mathematical notation, i.e., integer coordinates x and μ, between 0 and w - 1 and 0 and h - 1 respectively, corresponding to the coordinates of a pixel in the image of width w and height h, then the horizontal angle 9 satisfies in each pixel that:

[0072] 0(x,y) = ^(0(^-1,0)-0(0,0))+0(0,0)

[0073] Similarly, the vertical angle verifies in each pixel that:

[0074] ^(à; y) = ^-(^(0,^-1)-^(0,0))+^(0,0)

[0075] Once the image sector preprocessing step is complete, the processing module is configured to construct image segments. Each image segment is constructed from three image sectors, ideally without overlap except for the last sector if the number of image sectors is not a multiple of three. In some embodiments, an image segment is constructed from three image sectors with partial overlap between each image sector. Partial overlap is defined as an overlap in which the overlap of the image sectors is less than a total overlap. Generally, the image segment sizes can be 2160x1280 (Spynel X), 1536x640 (Spynel S), or 1920x512 (Spynel M).

[0076] In another embodiment, the processing module can be configured to construct a panoramic image of the surrounding scene from the acquired image sectors. In this case, the processing module is then configured to crop said panoramic image in image segments according to the invention.

[0077] The processing module is then configured to process the image segments separately using a processing algorithm, depending on the type of scene observed by the image sensor. Three types of scenes can be observed: a marine scene, a terrestrial scene, or a marine and terrestrial scene. The image segments are processed as images for the so-called "marine" and "terrestrial" processing algorithms. The processing module is therefore configured to process the image segments differently depending on the type of scene observed.

[0078] The processing module is configured to use so-called "maritime" processing when a maritime scene or part of a scene is observed by the image sensor.

[0079] The processing module is configured to use so-called "terrestrial" processing when a terrestrial scene or part of a scene is observed by the image sensor.

[0080] Figure [Fig. 2a] illustrates a flowchart of the processing applied by the processing module when a terrestrial scene is observed according to the invention.

[0081] The terrestrial processing is broken down into several steps. The pre-detection image, i.e., the input image segment, is converted using a multi-scale conversion to obtain an eight-bit image. For each image segment, the processing module is configured to:

[0082] - convert 2a the image segment according to a first, a second and a third conversion parameter, three output image segments are obtained and then averaged together,

[0083] - determine 2b an image of the differences from the image segment and a moving average of converted image segments,

[0084] - convert 2c the image of the differences according to a fourth conversion parameter,

[0085] - concatenate 2d the image segment and the image of the converted differences.

[0086] Step 2a is divided into sub-steps according to [Fig.2b].

[0087] The processing module is configured to determine Al three sets of parameters and for each set of parameters:

[0088] - determine a transformation A2 from the parameter set,

[0089] - apply the A3 transformation to the image segment.

[0090] The processing module is then configured to calculate (step A1 of [Fig. 2b]), for an input image, i.e., an image segment, local histograms for three local neighborhood sizes. The three sizes can correspond to the following, for example: size 1: 256x256, size 2: 128x64, size 3: 16x16. A local neighborhood size corresponds to a conversion parameter. Here, we therefore have three conversion parameters defined.

[0091] For the two largest local neighborhood sizes, these histograms are used by the processing module to retrieve predetermined quantiles (step A2 of [Fig. 2b]). In the case of the present invention, the processing module is configured to retrieve the minimum quantile, the maximum quantile, and the 1% and 99% quantiles. The processing module is then configured to apply a piecewise affine transformation to each neighborhood (step A3 of [Fig. 2b]). That is, the processing module is configured to linearly project onto the interval [0, 25.5] the pixels between the minimum value of the image and the 1% quantile of the image. The processing module is also configured to linearly project onto [25.5, 229.5] the values ​​contained between the 1% quantile and the 99% quantile, and the remainder onto [229.5, 255]. The processing module, in the case of the smallest size, is configured to perform a linear transformation from minimum to maximum towards [0, 255].

[0092] A floating-point image, that is, an image encoded with floating-point numbers, is obtained as output for each local neighborhood size. These images are then averaged for the three local neighborhood sizes and subsequently converted to eight-bit integers. The processing module is then configured to calculate a moving-average difference image (step 2b of [Fig. 2a]) at this position of the image before detection. To calculate the difference image, the processing module subtracts the moving average of the input images (i.e., the image segments before detection) from the input image segment (by a factor of 0.95).

[0093] The processing module is then configured to convert (2c in [Fig. 2a]) the difference image into a moving average using a simple scale conversion. That is, the conversion operates in the same way as explained previously but with a single neighborhood size. The processing module is configured to calculate local histograms on a local neighborhood size. Here, size 2 is used. Size 2 corresponds to the fourth conversion parameter, which, in this embodiment, is similar to one of the first three conversion parameters defined above. Once the image segment has been processed in this way, the processing module is configured to round the pixel values ​​to integers. The processing module can also be configured to then multiply the signal by a constant.

[0094] The resulting "current image" and "difference image" pair is concatenated (step 2d of [Fig. 2a]) by the processing module and then transmitted to the neural network for detection. The "current image" refers to the most recently available image segment, i.e., the input image segment for processing. A subtraction of one hundred and twenty-seven is performed before providing the image to the neural network. That is, the processing module is configured to center the signal, in this case by subtracting one hundred and twenty-seven.

[0095] A CLAHE terrestrial image can be seen by referring to [Fig. 3a] and, in [Fig. 3b], the first channel of the terrestrial image of [Fig. 3a] processed by the so-called terrestrial processing according to the invention.

[0096] The so-called "maritime" processing is configured to remove low-frequency information from the image, or image segment, by removing a heavily blurred version of the input image, the input image having a fourteen-bit encoding. The processing module is then configured to apply a piecewise affine transformation using the image quantiles. More specifically, the processing module is configured, with the following proposed parameters, to linearly project onto the interval [0, 25.5] the pixels between the minimum value of the image and the 1% quantile of the image. The processing module is also configured to linearly project onto [25.5, 229.5] the values ​​contained between the 1% quantile and the 99% quantile, and the remainder onto [229.5, 255].Once the image segment has been processed, the processing module is configured to round the pixel values ​​to integers and subtract 127 before providing the image to the neural network. This projection step corresponds to normalization before the neural network runs. In other words, the processing module is configured to center the signal, in this case by subtracting 127. The processing module can also be configured to then multiply the signal by a constant.

[0097] Referring to [Fig.4a] we can see a CLAHE maritime image and, in [Fig.4b], the maritime image of [Fig.4a] processed by the so-called maritime processing according to the invention.

[0098] In an embodiment where a maritime and terrestrial scene is observed by the image sensor, maritime processing as well as terrestrial processing can be used by the processing module 2.

[0099] When the image segments have been processed according to the type of scene observed, the processing module is configured to detect one or more objects in each processed image segment via a neural network.

[0100] Regarding the neural network, a network architecture based on a TTFNet (Training-Time-Friendly Network) is used. Any other compatible network type may be used in other embodiments. The neural network is configured to return an array, for example, of the one hundred highest-score detections. This allows detections below a user-defined threshold to be eliminated. The processing module can also be configured to use an additional algorithm to filter out duplicate detections. In one particular embodiment, the additional algorithm is a non-maximum suppression (NMS) algorithm. The detection step then allows the detection of objects present or absent in each image segment.

[0101] The processing module is then configured to construct a panoramic image of the scene from the image segments processed by the neural network. The processing module is finally configured to track said detected objects of the scene. constructed via a tracking algorithm in said image, the detected object(s) being associated with a tracking category.

[0102] Detections are matched with previous detections via the tracking algorithm. The detected objects are then classified into one of the following categories: alarm or trail. The objects are then filtered according to rules defined by the user and transmitted to them. The panoramic image can be displayed to the user, and the detected objects can be overlaid, for example, on this panoramic image.

[0103] Typically at least one of the means of the device according to the invention described above, preferably each of the means of the device according to the invention described above, is a technical means.

[0104] Typically, each of the means of the device according to the invention described above may include at least one computer, central processing unit or computing unit, analog electronic circuit (preferably dedicated), digital electronic circuit (preferably dedicated) and / or microprocessor (preferably dedicated) and / or software means.

[0105] Of course, the invention is not limited to the examples just described and many modifications can be made to these examples without departing from the scope of the invention.

[0106] Of course, the various embodiments, features, forms, and variants of the invention can be combined with one another in various ways, provided they are not incompatible or mutually exclusive. In particular, all the embodiments and variants described above are combinable with each other.

Claims

Claims

1. A method for detecting objects in a terrestrial scene, the method comprising the following steps: - acquiring a plurality of image sectors of a terrestrial scene by at least one image sensor (1), - preprocessing each of the image sectors to correct at least one image irregularity, - constructing image segments of the terrestrial scene, each image segment being constructed from at least two preprocessed image sectors, - processing the image segments by a terrestrial processing algorithm, - detecting objects in each image segment by a neural network, temporal information being provided to each image segment supplied to the neural network, - reconstructing the image of the terrestrial scene from said image segments.

2. The method of claim 1, wherein the at least one image sensor corresponds to a rotating image sensor or a fixed image sensor.

3. Method according to any one of the preceding claims, wherein the method also comprises the following step: - tracking the object(s) detected by a tracking algorithm in said image of the reconstructed scene, the detected object(s) being associated with a tracking category.

4. The method of claim 3, wherein tracking the detected object comprises matching the detected object of a detection N with the detected object of a previous detection Nl by the tracking algorithm.

5. Method according to any one of the preceding claims, in which the detected object(s) are associated with at least one of the following tracking categories: - alarm, - track.

6. A method according to any preceding claim, wherein the preprocessing step also comprises a step of transforming the image according to a geometric transformation.

7. Method according to any one of the preceding claims, in which the step of processing the image segments, when a terrestrial scene is observed, comprises the following steps, for each of the image segments: - calculates, for each image segment, local histograms on three local neighborhood sizes, - multi-scale conversion (2a) of each image segment according to a first, a second and a third conversion parameter, into three output image segments then the three image segments being averaged together, each local neighborhood size corresponding to a conversion parameter, - determination of a difference image (2b) from the image segment and a running average of the converted image segments, - single-scale conversion (2c) of the difference image according to a fourth conversion parameter, the fourth parameter corresponding to a local neighborhood size,- concatenation (2d) of the converted image segment and difference image.,

8. Method according to claim 7, in which the step of converting the image segment comprises the following steps: - determining (A1) three sets of parameters, each set of parameters corresponding to predetermined quantiles, For each set of parameters: - determining a transformation (A2) from the set of parameters, the transformation corresponding to a piecewise affine transformation, - applying the transformation (A3) to the image segment.

9. System for detecting objects in a terrestrial scene, the system comprising: - at least one image sensor (1) arranged to acquire several image sectors of an observed scene, - a processing module arranged and / or programmed to apply the method according to any one of claims 1 to 8.

10. A computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to any one of claims 1 to 8.

Images