Method for determining positions using an optronic system in a scene, optronic system and associated vehicle

Abstract

Method for determining positions by an optronic system in a scene, optronic system and associated vehicle. The present invention relates to a method for determining at least one position by an optronic system (18) in a scene (10), the method comprising: an input data acquisition phase consisting of an image of the scene (10) and a vignette corresponding to a portion of the aerial representation covering the scene (10); an image data processing phase; a vignette processing phase; a projection phase, in a common coordinate system, of segments seen from the ground and segments seen from the sky; a phase of associating segments seen from the ground and segments seen from the sky in the common coordinate system; and a phase of determining at least one position for the optronic system (18) based on the association performed. Figure for the abstract: 1

Description

Title of the invention: Method for determining positions by an optronic system in a scene, optronic system and associated vehicle

[0001] The present invention relates to a method for determining at least one position and at least one orientation by an optronic system in a scene. The present invention also relates to such an optronic system and an associated vehicle.

[0002] Collaborative combat, a hallmark of modern armed forces, aims to share the tactical situation with all actors on the battlefield through information enhancement. The first piece of information to be shared between actors is each actor's position. This position is maintained whether the actor is stationary or in motion.

[0003] Since the 1990s and the first Gulf War, the use of GPS (Global Positioning System) and more generally of GNSS (in English "Global Navigation Satellite System" translated into French as "Système Global de Navigation par Satellite", including GPS, GLONASS, GALILEO) has become widespread within NATO forces for positioning.

[0004] However, recent conflicts have shown that this type of satellite positioning system is susceptible to jamming, spoofing, or deception, preventing the use of GNSS solutions. This is referred to as "GNSS denied," meaning a denial of service to the use of satellite positioning. Furthermore, it has been demonstrated that it is possible to destroy a satellite in orbit, and therefore services relying on constellations of multiple satellites. Finally, GNSS systems do not function when signal strength becomes too weak, for example, inside buildings or under cover.

[0005] Furthermore, operations must be carried out with the greatest possible discretion in order to reduce adversary detection capabilities, which reduces or even prohibits the use of electromagnetic emissions. This electromagnetic restraint also contributes to extending the vehicle's range in terms of energy consumption.

[0006] We therefore seek to solve the problem of positioning in a contested environment. By contested environment, we mean an environment potentially without access to GNSS signals, without means of communication, without an electromagnetic emission sensor (vs. discretion and power consumption), without collaborating elements in the environment (beacons, specific markings, etc.) as in US 2007 / 0276590 A, and without using pre-existing images taken from the ground as in US 2012 / 0300020 A. Such images are indeed not available or updated in hostile areas (e.g., in enemy territory).

[0007] We therefore return to the use of classic navigation solutions, in particular by using a map and a manual or digital magnetic bearing compass, or by means of an inertial navigation system.

[0008] In particular, the use of a map and a manual or digital magnetic bearing compass remains the most basic method of determining one's position. It requires exiting the vehicle to avoid biasing the field measurement with the vehicle's metallic mass and also demonstrates a certain level of technical skill. This method is therefore lengthy, tedious, and risky, as it involves exposure to potential external threats, and furthermore, it remains rather imprecise (positioning accurate to within a few tens of meters, and the use of Magnetic North requires declination corrections).

[0009] The use of an inertial navigation system (INS) allows, after initial positioning, the maintenance of a current position by measuring all the vehicle's movements. This system nevertheless requires regular recalibration by an external means (typically GPS recalibration) to reduce drift. However, in practice, the crew of a military vehicle is preoccupied with its protection and coordination with other vehicles. Recalibration is therefore not carried out systematically, or in most cases, not at all.

[0010] There is therefore a need for an optronic system that can determine positions more precisely and robustly in a hostile environment, limiting the demands on the crew, without GNSS signals, without the emission and reception of electromagnetic waves emitted from objects or a third-party network, and without emitting radar or lidar waves to increase its stealth and energy autonomy.

[0011] To this end, the present description relates to a method for determining at least one position by an optronic system in a scene, the scene comprising ground features with known geographical coordinates, such as roads or paths, the optronic system comprising the following elements integrated into the optronic system: - an imaging device capable of acquiring images of the scene as seen from the optronic system, - a memory in which is stored a bird's-eye view representation of an area of ​​space including at least the scene, with ground elements georeferenced to the bird's-eye view representation, and - a unit of calculation,

[0012] the method being implemented by the optronic system and comprising: - an input data acquisition phase, the input data including: • an imaging data of the scene, the imaging data being an image or a stream of images of the scene seen from the optronic system, the imaging data having been acquired by the imaging device, • a thumbnail corresponding to a portion of the aerial view of the scene, the thumbnail having been extracted from memory by the processing unit, an imaging data processing phase implemented by the computing unit, comprising: • the segmentation of the imagery data so as to highlight elements belonging to predefined classes on the imagery data, the predefined classes including at least one class representative of ground elements and classes representative of other elements of the scene, at least one of the highlighted elements being a ground element, • the determination of contours, on the image data, corresponding to the intersections between each highlighted ground feature and the highlighted features for other classes, • the extraction of segments, known as ground-view segments, representative of each ground element highlighted in the image data, based on the contours determined for said ground element, a thumbnail processing phase implemented by the computing unit, comprising: • Segmenting the thumbnail so as to highlight at least one element on the ground, • Determining the outlines of each ground element highlighted in the thumbnail, • the extraction of segments, known as aerial view segments, representative of each ground element highlighted in the thumbnail, based on the contours determined for said ground element, a projection phase, by the computing unit, of segments seen from the ground and segments seen from the sky in a common coordinate system, the common coordinate system being the coordinate system of the image data or the coordinate system of the thumbnail, a phase of association, by the unit of calculation, of segments seen from the ground and segments seen from the sky in the common frame of reference, and a phase of determination, by the computing unit, of at least one position for the optronic system as a function of the association made.

[0013] According to other advantageous aspects, the process comprises one or more of the following features, taken individually or in all technically possible combinations:

[0014] - the images acquired by the imaging device are panoramic or quasi-panoramics of the scene, the image data being a panoramic or quasi-panoramic image or a stream of panoramic or quasi-panoramic images of the scene.

[0015] - the optronic system is in motion in the scene, a previous position of the optronic system in the scene being stored in memory;

[0016] - the association phase includes the determination of a transformation corresponding to the association between segments seen from the ground and segments seen from the sky in the common frame of reference, the phase of determining at least one position for the optronic system including the determination of an approximate position for the optronic system as a function of at least the previous position and the determined transformation;

[0017] - each ground element belonging to a ground class is a linear element comprising two ends, at the end of the association phase, each end of each ground element considered on the image data is described by two pairs of coordinates, one of the pairs of coordinates being the pixel coordinates of said end on the image data, and the other pair of coordinates being the geographic coordinates of said end on the thumbnail, the phase of determining at least one position for the optronic system including the determination of an absolute position and an absolute orientation as a function of the pairs of coordinates determined for each ground element;

[0018] - the association phase comprising the association of the segments seen from the ground and the segments seen from the sky corresponding to the same directions;

[0019] - the association determines a transformation such that after application to the elements on the ground, the difference between the segments seen from the ground and the segments seen from the sky is minimal compared to other possible associations;

[0020] - the ground elements are chosen from one or more of the following elements: - streets, alleyways, avenues, or boulevards in urban areas, - highways, national roads, departmental roads in urban, suburban or rural areas, - agricultural, forestry, and mountain roads in rural areas, - roads and tracks in desert areas, and - fields, or open uncultivated areas in rural areas;

[0021] - the method includes an initialization phase consisting of determining a position initial and an initial orientation of the optronic system;

[0022] - the optronic system is deployed on a platform jointly with a navigation instrument, such as an inertial navigation system or inertial measurement unit, the process including a phase of recalibrating the navigation instrument according to a pose determined by the optronic system;

[0023] - the segmentations performed on the image data and on the thumbnail are put implemented by a deep learning algorithm, preferably a semantic segmentation algorithm;

[0024] - the common frame considered during the projection phase is the data frame imaging;

[0025] - the imaging device comprises several optronic sensors distributed around of the vehicle, the imaging data being constructed from images captured by the optronic sensors, said images being dated and assembled using beam fitting processing to establish the imaging data.

[0026] This description further relates to an optronic system for determining at least one position by means of an optronic system in a scene, the scene comprising ground features with known geographical coordinates, such as roads or paths, the optronic system comprising elements integrated into said optronic system and configured to implement a method as described above, the integrated elements comprising at least the following elements: - an imaging device capable of acquiring images of the scene as seen from the optronic system, - a memory in which is stored a bird's-eye view representation of an area of ​​space including at least the scene, with ground elements georeferenced to the bird's-eye view representation, and - a unit of calculation.

[0027] This description also relates to a vehicle, such as a land vehicle, comprising an optronic system as described above.

[0028] The invention will become clearer upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0029] [Fig-1] [Fig.1] is a schematic representation of an example of a scene including reference elements (landmarks), particularly ground-based elements, with an optronic system integrated into a vehicle also present on the scene,

[0030] [Fig.2] [Fig.2], a schematic representation of an example of a system optronics comprising elements integrated into said system

[0031] [Fig.3] [Fig.3], a flowchart of an example of the implementation of a method for determining at least one position in a scene,

[0032] [Fig.4] [Fig.4], a schematic representation of an example illustrating the three steps of a phase of processing image data, namely segmentation (into semantic classes) of the image data, determination of contours and extraction of segments seen from the ground, and

[0033] [Fig.5] [Fig.5], a schematic representation of an example illustrating a contour determination step for a ground element highlighted on a thumbnail.

[0034] In the following description, the absolute (geographic) orientation of an object in a scene is defined as the vector joining the system to the object; the associated direction being the line containing this vector. This absolute orientation is characterized by angles expressed with respect to a geographic reference. The most commonly used are the azimuth angle, which expresses the orientation in a locally horizontal plane (tangent to the ellipsoid associated with the geoid) with respect to the local geographic meridian, and the elevation angle (or inclination angle), which expresses the orientation in a vertical plane with respect to the locally horizontal plane. A compass is typically used to measure azimuth. An inclinometer is typically used to measure elevation.A relative orientation is defined with respect to another reference orientation (i.e., an angular difference between two orientations), characterized by the bearing angle in the horizontal plane and the elevation angle in the vertical plane. A goniometer typically allows the measurement of bearing and elevation.

[0035] The attitude of an element is defined as the information that allows the element to be fully oriented in a frame of reference covering the 3 dimensions of geographic space (for example, in a minimal way with the 3 Euler angles of roll, pitch and yaw).

[0036] Finally, the term “segment” refers to a straight line segment.

[0037] A scene 10 is illustrated by way of example in [Fig. 1]. A scene designates a The theater of operations is the place where an action takes place. The stage is therefore an extensive space with sufficient dimensions to allow the action to unfold. The stage is typically an outdoor space.

[0038] Scene 10 includes reference elements, also called landmarks or reference structures, having known geographical coordinates and precision.

[0039] Each reference element is a fixed and notable object or feature of the scene 10. The coordinates (latitude, longitude) of each reference element are known. Optionally, the altitude of each reference element is also known. Preferably, the accuracy of the longitude and latitude coordinates of each reference element is known from the quality metadata attached to an orthophoto representing the scene from an aerial view. Similarly, the altitude accuracy is known from the metadata of a digital terrain model indicating the altitudes of ground points in the scene used in the vicinity of the element. reference. Regarding the altimetric scene model, we emphasize that the system implementation preferably aims to avoid constraining this data source by requiring a very precise and dense spatial description (such as, for example <lm). Cette contrainte d’usage est posée pour limiter les contraintes sur l'acquisition des données de référence, soit du fait de leur vieillissement, soit de leur simple disponibilité. Les contraintes posées concernant l’usage des données de référence conduit alors à privilégier l’usage d’éléments de référence situés au sol et présentant une faible hauteur de sursol.

[0040] Reference elements are, for example, points belonging to the following: a construction (building, bell tower, lighthouse, road, bridge, etc.) whose coordinates can be found on an ortho-image of satellite or airborne origin or a topographic map and a natural element (mountain, rock, hilltop, vegetation, tree, etc.) whose coordinates can be found on an ortho-image of satellite or airborne origin.

[0041] At least some of the reference elements are ground elements 12, such as ground tracks. More specifically, the ground elements 12 are, for example: - streets, alleyways, avenues, or boulevards in urban areas, - highways, national roads, departmental roads in urban, suburban or rural areas, - agricultural, forestry, and mountain roads in rural areas, - roads and tracks in desert areas, and - fields, or open uncultivated areas in rural areas.

[0042] In the example illustrated by [Fig.1], the reference elements are pylons and ground elements 12, in particular roads.

[0043] To implement the method according to the invention, an optronic system 18 is located in the scene 10. The optronic system 18 is therefore also an object of the scene 10.

[0044] The optronic system 18 is preferably mounted on a land vehicle 19 as in the example of [Fig. 1]. The vehicle is, for example, of a military type, such as an assault tank or an infantry fighting vehicle (IFV). Such a military vehicle is particularly suited to carry a plurality of weapons and to protect the operator(s) installed inside the vehicle.

[0045] Advantageously, a navigation system, such as an inertial component, for example an inertial measurement unit (IMU), is also mounted on the vehicle. Preferably, this inertial component provides a solution (which will only be approximate due to drift), but is nevertheless used as an initial solution for determining the pose. The quality of this initial pose (position and attitude) depends on the class or intrinsic quality of the IMU, its accuracy, and the time elapsed since its last measurement. Recalibration. In practice, a lower cost inertial component will lead to a faster drift of its solution and will require more frequent recalibration to maintain a pre-set navigation quality setpoint.

[0046] In one embodiment, the optronic system 18 has a geometric image capture model. This image capture model makes it possible to associate a pixel of one of the images from the optronic system 18 with a spatial direction described by two angles (bearing, elevation) in the coordinate system of the frame attached to the optronic system 18. This frame of the optronic system 18 has an attitude relative to the local geographic coordinate system, and the three (Euler) angles characterizing this attitude constitute three of the six unknowns of the system's pose to be determined. Thus, any point of a reference element of the scene having coordinates in an optronic image has a relative direction in the frame of the optronic system 18.

[0047] The optronic system 18 comprises elements integrated into said optronic system 18. By the term "integrated", it is understood that the elements are physically and software incorporated into said optronic system 18. Such elements therefore form a single block in the optronic system 18.

[0048] The elements integrated into the optronic system 18 shown in [Fig.2] include at least the following elements: an imaging device 20, a memory 22 and a computing unit 28. Optionally, the optronic system 18 also includes a visualization element 24 and an attitude measurement module 30.

[0049] The imaging device 20 (also referred to as the "sensor" in the description) is designed to acquire images of the scene 10. The native images acquired by the imaging device 20 can be resampled in a specific geometry or projection. The images are composed of pixels identified by pixel coordinates. Each pixel corresponds to a direction characterized by its angular values ​​(bearing, elevation) in a frame of reference linked to the optronic sensor (imaging device 20). The sensor's mounting on the vehicle is, for example, determined by a calibration procedure, which allows the angular information from a direction in the sensor's frame of reference to be transferred to a frame of reference of the vehicle. In the case of several distributed sensors, the sensor mounting allows the directions recorded by each sensor to be processed in a common frame of reference of the vehicle.

[0050] Preferably, the images acquired by the imaging device are panoramic or quasi-panoramic images. An image is said to be panoramic when it images a scene over 360° in azimuth. The elevation range is, for example, between 75° and -15°. A quasi-panoramic image is a panoramic image exhibiting occultation zones. An occultation zone is an angular sector in azimuth for which the scene is not imaged. An occultation zone extends for example over a maximum of 25° in bearing and over the entire elevation range of the image.

[0051] In practice, the elevation ranges may be limited by objects in the image. Image processing can further reduce the areas actually acquired when these correspond to areas of low a priori information, such as elevated sites corresponding to the sky, for example. In the example of [Fig. 1], on vehicle 19, the sensor 18 is located a few meters above the ground, and the objects in the vicinity of the vehicle have the best image resolution.

[0052] Advantageously, the imaging device 20 is suitable for operating in several spectral bands, for example, in the visible and in the infrared.

[0053] The imaging device 20 is, for example, formed by a single sensor (camera). This allows for optimal spatial and temporal coherence for the acquired images.

[0054] Alternatively, the imaging device 20 is formed by a set of sensors (cameras) and their images are then time-synchronized (dated) and geometrically adjusted in order to extract useful information as if one had an instantaneous view of the scene (where applicable panoramic).

[0055] Preferably, the imaging device 20 is an omnidirectional imaging device, that is to say an imaging device capable of providing images over 360° in azimuth and up to 180° in elevation.

[0056] In another variant, the imaging device 20 does not image the entire 360° due to masking or an architecture composed of a set of non-overlapping distributed cameras. As mentioned previously, the images are then described as quasi-panoramic.

[0057] In one example, the imaging device 20 includes one or more MINERVA sensors developed by THALES.

[0058] Data is stored in memory 22. The data includes, in particular, a bird's-eye view of an area of ​​space comprising at least scene 10. The bird's-eye view is, for example, an orthophoto, such as a satellite image or an aerial image. Alternatively, the bird's-eye view is a map in the form of a digital image, for example, a topographic or symbolic map (of the type produced by the IGN for the Institut Géographique National in France). Alternatively, the bird's-eye view is a combination of an image and a map.

[0059] The reference elements, and in particular the ground elements 12, of the scene 10 are georeferenced on the aerial view representation. Preferably, each point (pixel, element) of the aerial view representation is associated with geographic coordinates.

[0060] The geographical coordinates of the reference elements, and in particular the ground elements 12, are, for example, expressed by a latitude value, a longitude value, and optionally an altitude value. The accuracy errors associated with these data are preferably also provided.

[0061] Preferably, memory 22 includes ephemerides giving the position of the sun and other bodies of the solar system according to the dating.

[0062] The computing unit 28 is suitable for receiving data from other elements integrated into the optronic system 18, in particular images from the imaging device 20, data stored in the memory 22 and measurements made by the attitude measurement module 30, or from external navigation instruments, for example, on board the vehicle on which the optronic system 18 is mounted.

[0063] The computing unit 28 is, for example, a processor.

[0064] In one example, the computing unit 28 interacts with a computer program product that includes an information carrier. The information carrier is a carrier readable by the computing unit 28.

[0065] The readable information medium is a medium suitable for storing electronic instructions and capable of being connected to a bus of a computer system. For example, the readable information medium is an optical disc, a CD-ROM, a magneto-optical disc, a ROM, a RAM, an EPROM, an EEPROM, a magnetic card, an optical card, or a USB flash drive. The computer program product, comprising program instructions, is stored on the information medium.

[0066] The computer program can be loaded onto the computing unit 28 and leads to the implementation of a method for determining at least one position in a scene 10, when the computer program is implemented on the computing unit 28 as will be described later in the description.

[0067] The display element 24 is suitable for displaying images from the imaging device 20 and / or data stored in memory 22, such as the aerial view representation.

[0068] The display element 24 is, for example, a display, such as an OLED screen.

[0069] The attitude measurement module 30 is optional. The attitude measurement module 30 is designed to perform measurements relative to the reference elements 12 of the scene 10.

[0070] Preferably, the attitude measurement module 30 includes at least one element, such as a sensor, selected from: a goniometer, a rangefinder, an inclinometer and a dating system (millisecond, second, day, month, year).

[0071] The operation of the optronic system 18, which involves implementing a method for determining at least one position in a scene 10, will now be described with reference to the flowchart in [Fig. 3]. The determination method is implemented by the elements integrated into the optronic system 18.

[0072] Preferably, the different phases of the process are carried out continuously (in real time), for example at the acquisition frequency of the imaging device 20.

[0073] Preferably, the optronic system 18 is in motion in the scene 10 and a previous position (possibly also an orientation) of the optronic system 18 in the scene 10 has been stored in memory 22.

[0074] The previous position is, for example, an approximate position or an absolute position.

[0075] The previous position was, for example, obtained by a GNSS device (possibly in degraded operation), or during a maintenance of the position and attitude of the optronic system 18 by an inertial unit on board the vehicle or in the optronic system 18, or by a previous approximate calculation.

[0076] In one embodiment, the method includes an initialization phase consisting of determining an initial position and orientation (initial pose) of the optronic system 18 (at the start of the mission or when the vehicle departs). For example, this initial pose is provided either by GNSS (if it is reliable at the mission start location), by image geolocation, by bearings from georeferenced landmarks (by the armed forces' geographic service), or by any other static positioning method. This initialization step will also benefit the alignment of the CNI. INPUT DATA ACQUISITION (PHASE 100)

[0077] The determination process includes a phase 100 of input data acquisition.

[0078] The input data includes image data from scene 10 and a thumbnail.

[0079] The imaging data is an image or a stream of images of the scene 10 seen from the optronic system 18. The imaging data was acquired by the imaging device 20.

[0080] Preferably, the image data is a panoramic (or quasi-panoramic) image, or a stream of panoramic (or quasi-panoramic) images of scene 10.

[0081] Acquisition can take place with the carrier stationary or in motion. The speed of movement must be compatible with the integration time of the imaging device 20 to avoid introducing blur into the image. For example, a speed on the order of 50 km / h can be reached with an integration time of a few milliseconds. Optionally, if the aircraft is equipped with an inertial measurement unit (IMU), the position maintained by the IMU will be acquired simultaneously with the image acquisition.

[0082] In one example, the image data is constructed from videos of optronic sensors distributed around the vehicle. The images from said sensors are time-stamped and assembled using beam-fitting processing to establish a panoramic image or a stream of panoramic images of the scene.

[0083] Knowledge of the image arrangement, describing their position and attitude in a common reference frame to the vehicle, can be refined as needed to meet a performance objective. For this purpose, preference is given to using observations consisting of: - Matching of image primitives in the overlap areas between images; to improve accuracy, small point primitives with sub-pixel accuracy are used. - Matching objects with known ground coordinates with their corresponding image coordinates; - A mixture of the two previous types of observations, relative and absolute.

[0084] The thumbnail corresponds to a portion of the aerial view representation covering scene 10. The thumbnail was extracted from memory 22 by the processing unit 28.

[0085] For example, the extracted thumbnail is centered on the previous position of the optronic system 18. IMAGING DATA PROCESSING (PHASE 200)

[0086] The determination process includes a phase 200 of processing the imaging data by the computing unit 28.

[0087] The processing phase 200 comprises segmenting the image data so as to highlight, in the image data, elements belonging to predefined classes. The predefined classes include at least one class representing ground elements 12 (e.g., roadways) and classes representing other scene elements 10. At least one of the highlighted elements is a ground element 12.

[0088] The representative classes of other elements of the scene are, for example, classes corresponding to other reference elements. For example, these classes correspond to buildings or natural features. Alternatively, these classes correspond to non-reference elements, such as a vehicle or a living being.

[0089] In one example implementation, segmentation is performed by a deep learning algorithm. Preferably, segmentation is performed by a semantic segmentation algorithm.

[0090] An example of segmentation is illustrated in the image at the top of [Fig.4].

[0091] The processing phase 200 includes the determination of contours, on the imaging data, corresponding to the intersections between each highlighted ground element 12 and the highlighted elements for other classes.

[0092] An example of edge detection is illustrated in the middle image of [Fig.4]. In particular in this figure, the edges correspond to the intersection between the ground track (highlighted by the ground class) and another class (field, forest...).

[0093] The processing phase 200 includes the extraction of segments, called ground-view segments, representative of each ground element 12 highlighted on the imaging data, according to the contours determined for said ground element.

[0094] For example, the segments seen from the ground are extracted using an Edge Drawing Lines (EDLINES) or Line Segment Detector (LSD) type segment extraction algorithm in its classic or AI (LSDNet) version or, by way of non-limitation, variants such as LB-LSD in order to improve the rendering of contours in segments.

[0095] For example, for segments seen from the ground, we will consider the straight line segments located at the interfaces between: - Road (path, motorway, paved road) and more generally on the ground (for classes such as sidewalk, lawn, cultivated land, etc.), - Road and building (corresponding to the base of the buildings), - Ground and building (corresponding to the base of the buildings), - Soil and forest, - Road and forest (forest edge), - Ground and pylon (base of the pylon), and - Road and pylon.

[0096] An example of the extraction of segments seen from the ground is illustrated in the image at the bottom of [Fig.4],

[0097] Due to the sensor's low height of a few meters above the ground, the resulting low-angle shooting conditions, and the sensor's limited resolution, scene elements located close to the sensor are most likely to be visible and easily recognizable in the aerial view. This area, extending over a few tens of meters, is preferably the preferred zone for selecting elements to be combined between the two views: the optronic system view (ground view) and the aerial view. VIGNETTE PROCESSING (PHASE 300)

[0098] The determination process includes a step 300 of processing the vignette by the computing unit 28.

[0099] The processing phase 300 includes the segmentation of the thumbnail (after selection of the area to be processed for the thumbnail) so as to highlight on the thumbnail at least one ground element 12.

[0100] In an example implementation, the segmentation is performed in the same way as for the imagery data, i.e., by segmenting the thumbnail so as to highlight elements belonging to predefined classes on the thumbnail. At least one of the highlighted elements is a ground element 12.

[0101] In one example implementation, segmentation is performed by a deep learning algorithm. Preferably, segmentation is implemented by a semantic segmentation algorithm. Preferably, the semantic segmentation algorithm is applied to color (RGB) orthophotos to extract the network of roads and paths as a grayscale image, the level of which represents the confidence level of belonging to the "roads and paths" class. In particular, for example, the network used for this processing phase of ground-view images does not share the architecture or weights of the network used for extracting roads from aerial-view images. The two networks are independent and can work on images of different spectral types, with distinct constraints in terms of the number of pixels to be processed and the inference time.

[0102] Another approach is to use digitized topographic maps. Starting from these maps, it is still possible to extract contours in the form of segments using the same processing methods as those used for imagery data.

[0103] Another approach would be to directly use a vector database representing roads, but this topographic data is not available for all areas of our planet with the necessary quality. Furthermore, when produced, it does not necessarily incorporate the level of detail for small paths that could be extracted from orthophotos.

[0104] The processing phase 300 includes the determination of contours for the ground elements 12 highlighted on the thumbnail. In an example of implementation, the contour determination is carried out in the same way as for the image data, i.e. by determining contours, on the thumbnail, corresponding to the intersections between each highlighted ground element 12 and the highlighted elements for other classes.

[0105] The processing phase 300 includes the extraction of segments, called sky-view segments, corresponding to the contours determined for each ground element 12 considered.

[0106] For example, segments are extracted using a segment extraction algorithm such as LSD (Line Segment Detection), insofar as it is also used for segments extracted during segmentation from the data of imaging. An example of edge detection on a topographic map is illustrated in [Fig.5]. PROJECTION (PHASE 400)

[0107] The determination process includes a phase 400 of projection, by the computing unit 28, of the segments seen from the ground and the segments seen from the sky in a common reference frame.

[0108] The common reference frame is that of the image data or that of the thumbnail or any other intermediate representation.

[0109] Preferably, the common reference frame considered during the projection phase is the thumbnail reference frame.

[0110] Alternatively, the common reference frame is the reference frame of the imaging data. ASSOCIATION (PHASE 500)

[0111] The determination process includes a phase 500 of association, by the calculation unit 28, of the segments seen from the ground and the segments seen from the sky in the common frame of reference.

[0112] In one embodiment, the association phase comprises the association of segments seen from the ground and segments seen from the sky corresponding to the same directions. In particular, the paired line segments belong to the same line but generally have different endpoints.

[0113] Preferably, each ground element 12 belonging to a ground class is a linear element comprising two endpoints. At the end of the association phase, each endpoint of each ground element 12 considered on the image data is described by two pairs of coordinates. One pair of coordinates corresponds to the pixel coordinates of said endpoint on the image data, and the other pair of coordinates corresponds to the geographic coordinates of said endpoint on the thumbnail.

[0114] In one embodiment, the determined association is such that the difference between the segments seen from the ground and the segments seen from the sky is minimal compared to other possible associations. The determined association is therefore the best association.

[0115] Preferably, the association phase comprises: - a step of selecting the contours of eligible elements of the thumbnail based on the approximate position of the system and favoring the neighborhood zone located around and near the sensor. The selection of elements in the imagery data is quite natural if we limit ourselves to the contours of the road class, but we can also limit the distance of the selected elements by restricting ourselves to the lowest sites in the imagery data, represented in the lower parts of [Fig.4]. - a step to determine the parameters of a transformation allowing the best superposition of the segments seen from the ground and the segments seen of the sky projected onto the common coordinate system. In particular, the transformation is a geometric transformation that allows us to go from the coordinate system of the image data to the coordinate system of the representation seen from the sky, or vice versa. A simple transformation, in terms of the number of parameters to estimate, is recommended for this step (for example, a 2D translation with possibly one rotation angle). The estimation of these parameters is preferably carried out by implementing at least one of the following tools: a Hough transform, a RANSAC algorithm (RANdom SAmple Consensus of Fischler & Bolles 1981), and a correlation. - a step of effectively matching the segments extracted from the thumbnail and the image data, performed using the previous transformation by selecting the most relevant segment from a window among the few potentially compatible segments of this transformation. A segment of the thumbnail can be associated with one or more segments of the image data, and conversely, a segment of the image data can be associated with one or more segments of the thumbnail. If 1 segment from one space is associated with N segments in the other space, then we have N matches or N observations.

[0116] DETERMINATION OF POSITIONS (AND ORIENTATIONS) (PHASE 600)

[0117] The determination process includes a phase 600 of determination, by the computing unit 28, of at least one position for the optronic system 18 as a function of the association made.

[0118] In one example of implementation, the determination phase 600 includes determining an approximate position for the optronic system 18 based on the previous position and the determined transformation. Thus, the transformation determined during the association step is directly used to correct the previous position, thereby obtaining an approximate position for the optronic system 18.

[0119] In addition, or alternatively, the determination phase 600 includes the estimation of an absolute position and an absolute orientation as a function of the pairs of coordinates determined for each associated ground element 12.

[0120] For the installation estimation, all associations between ground elements 12 are considered; with a minimum that depends on the type of estimator used: - With a Bayesian estimator, a single matching association allows the approximate pose solution to be inferred whether it is iterated (Kalman type) or batch since then as many constraint equations as parameters to be estimated are added to the matching observation equation to describe the information on the initial pose. - With a classical or weighted least squares estimator, based on minimizing a metric providing two observation equations by matching two segments, three matching associations are necessary to refine the approximate pose solution.

[0121] In particular, for example, the approximate pose (position and attitude) of the optronic system 18 can be obtained with a measuring device, such as an inclinometer for roll and pitch, a magnetic compass for azimuth, or more generally, more directly and more precisely, using the inertial measurement unit. The approximate pose allows the selection of candidate ground elements for matching and their projection onto the common frame of reference in order to find the transformation that best associates them with the set of candidates.Approximate posing also allows linearizing the observation equations in order to find an optimal pose by searching for pose increments, as a solution to a linear system to be solved iteratively; or to initialize the state vector and its covariance in a pose estimation approach using a Kalman-type filter.

[0122] Preferably, the determination phase 600 also includes the determination, by the computing unit 28, of the accuracy of the determined position, and where applicable, of the orientation. The accuracy is determined based on the accuracy errors associated with the geographic coordinates of the ground elements 12, and the errors related to the extraction of information from the imagery data. Typically, the overall uncertainty is obtained by propagating the uncertainties in an additional step for a batch approach that processes the associations simultaneously as a set, or it is calculated explicitly at each update of the filtering of an association in a Kalman-type approach.

[0123] For an optimal installation solution or estimated installation state, it is always possible to improve the installation obtained by means of additional observations. For example, the azimuth of the orientation obtained can be improved as follows: - The position of the sun is automatically determined in the image data to provide its direction (bearing, site) as a sensor reference. - We use the date of shooting, ephemerides integrated into the data 22 and the position of the sensor obtained in order to determine the reference direction of the sun in the local geographical frame. - We introduce the new association consisting of the correspondences between these 2 directions and the associated precisions in the estimator so that it improves in particular its azimuth positioning.

[0124] For reference, the center of the sun can be extracted sub-pixelally with a quality of class 0.2 mils. Errors related to other contributions are small. In comparison, this type of observation allows for a better quality of orientation than a high-end control unit.

[0125] Preferably, the determination phase 600 also includes the determination of at least one location of an object in the scene 10 by the optronic system 18 as a function of the position of the optronic system 18 determined by the previous phases of the method, the direction of the object according to its coordinates in the image data of the optronic system 18 and the distance by the rangefinder of the attitude module 30.

[0126] In a preferred approach exploiting specific elements complementary to a blind use of roadside contours, the association of specific elements such as intersection points or circular elements viewed from above is simple to associate unambiguously due to the limited order of the navigational error (approximately 10m) and the rarity or low spatial density of these particular structures; this allows: - to perform a recalibration on unambiguous structures; - to have an element capable of calculating a 3D pose via a lifting: • With two reference positions or point landmarks, a simplified bearing procedure is used. This simplification is achieved by using the azimuth of the navigation solution and knowledge of the sensor's mounting on the vehicle to deduce the bearing and solve a system of linear equations that gives the sensor's position in the local plane. A local geographic coordinate system (RGL) is used with the (approximate) navigation position, the current navigation solution, as its origin. Furthermore, a local plane is estimated using the elevations of the DEM at the values ​​of nodes near the origin. If the plane is too steep (typically ~10°), the bearing position is calculated by rectifying the measurements in this local plane. The landmark coordinates are projected onto the plane, and the bearing direction is deduced from the attitude of the current navigation solution and the sensor mounting.Using the bearing deduced from the navigation solution, the bearing equations provide a sensor position in the plane whose deviation from the origin corresponds to the correction to be applied to the current navigation position. In practice, the geodetic coordinate can be calculated, and the filter inferred from this position, along with the two directions associating the coordinates of the two landmarks with their bearing values ​​in the image. In the case of flat terrain, the calculated plane position is supplemented with the altitude. The position is interpolated into the DEM with the calculated position increased by the sensor height above ground. An error is calculated on the position based on the errors in the observations due to covariance propagation. • With three reference positions, a standard bearing procedure is used to calculate the sensor's planar position. As above, the planar position is supplemented with an altitude and position errors. Since the bearing is also calculated, the sensor mounting allows for the deduction of an azimuth for the vehicle's direction and an associated error. For a vehicle following a straight trajectory, the azimuth value can be compared to the direction of the wheel rim segments. - To estimate a landing solution using abundant information: • With more than 3 landmarks, an adjustment procedure can be used to estimate the planar position. The information is supplemented as above; in addition to improving quality, this approach allows for estimating the bearing and therefore the azimuth of the vehicle. The position obtained in RGL directly provides the spatial navigation correction. • With point landmarks and roadside segments matched, it is possible to estimate a 6D pose and its accuracy by its variance (as in the Visioloc3 patent). - By using a Bayesian estimator, more precisely a least-squares estimation constrained by the prior pose information provided by the current solution, the constraints allow us to increase the number of observation equations by as many equations as there are parameters to estimate. The constraint equations are constructed with the approximate values ​​of the parameters and their variance; the set of observation and constraint equations weighted by their variance allows us to infer the approximate solution and calculate its new variance.

[0127] Preferably, at least the determined position of the optronic system 18 is stored in the memory 22 of the optronic system 18 for later reuse, for example in the event of a failure of a navigation instrument. This can also be used for debriefing, evaluation, or fine-tuning phases during or after the mission. NAVIGATION INSTRUMENT RECALIBRATION (PHASE 700)

[0128] Optionally, when the optronic system 18 is deployed on a platform in conjunction with a navigation instrument, such as an inertial navigation system, the The process includes a phase 500 of recalibrating the navigation instrument according to the pose determined by the optronic system 18 (position and attitude approximated, optimized or updated, preferably the last pose obtained).

[0129] Alternatively, the recalibration is carried out based solely on the position of the optronic system.

[0130] When the platform is in motion, the inertial measurement unit will be recalibrated for example by the error between the position calculated by the process and the position maintained by the inertial measurement unit acquired simultaneously with the acquisition of the imaging data, as described in step 100. CONCLUSION

[0131] In conclusion, the present method makes it possible to use a completely autonomous image-based geographic positioning system, that is to say, independent of an external means such as an inertial measurement unit, GNSS, or rangefinder. This method can be implemented automatically without operator intervention.

[0132] The present method thus makes it possible to provide an autonomous, completely discreet (passive), and robust navigation solution for a vehicle equipped with an optronic sensor operating in a contested electromagnetic environment, without GNSS signals and in a short time. The method can also provide a performance prediction on the obtained poses, which is also useful for determining the localization accuracy of an object accessible using optronic information. Therefore, the method makes it possible to locate targets in contested environments.

[0133] In particular, the present method includes an automatic search step on images having different views (ground-level view and orthographic view of the sky) for elements having particular orientation properties which will allow them to be projected unambiguously into one or the other of these two spaces.

[0134] To solve this complex problem, in a preferred embodiment, semantic segmentation of the scene in both the aerial and ground views automatically allows, without the need for an operator, the classification of scene elements, including those located on the ground (roads), from those likely to pose a problem (vegetation, sky, buildings, etc.) since they present different appearances between the two views. These different appearances are notably in terms of parallax for vertical structures, age for vegetation, or simply invisibility from one of the views (e.g., the terrace of a building visible from the aerial view is not visible from the ground view).

[0135] In a preferred embodiment, being obtained via a vehicle located a few meters above the ground, the imagery data will contain at least a small amount of ground features (road, sidewalk, path) seen from 'above'; the roofs of buildings not being visible, as explained previously. On the other hand, it presents Clearly, building facades or trees are visible in the frontal view, whereas these elements are not visible in the orthographic image viewed from above. The ability to automatically select the few common elements in both views allows them to be projected in isolation in order to be robustly associated, having eliminated other elements that would produce ambiguities when combining all the elements.

[0136] In the preferred mode using specific observations, the filter is fed with the previously calculated partial pose information as well as directions (site, bearing) on ​​the point landmarks in sensor reference frame to feed the navigation filter.

[0137] Those skilled in the art will understand that the order of the different phases is given by way of example. For example, processing phase 200 can be reversed with processing phase 300.

[0138] A person skilled in the art will understand that the embodiments described above can be combined to form new embodiments provided that they are technically compatible.

Claims

Demands

1. A method for determining at least one position by an optronic system (18) in a scene (10), the scene (10) comprising ground features (12) with known geographic coordinates, such as roads or paths, the optronic system (18) comprising the following elements integrated into the optronic system (18): - an imaging device (20) suitable for acquiring images of the scene (10) as seen from the optronic system (18), - a memory (22) in which is stored a bird's-eye view representation of an area of ​​space comprising at least the scene (10), with ground elements (12) being geo-referenced to the bird's-eye view representation, and - a calculation unit (28), the process being implemented by the optronic system (18) and comprising: - an input data acquisition phase, the input data including: • an imaging data of the scene (10), the imaging data being an image or a stream of images of the scene (10) seen from the optronic system (18), the imaging data having been acquired by the imaging device (20), • a thumbnail corresponding to a portion of the aerial view representation covering the scene (10), the thumbnail having been extracted from memory (22) by the processing unit (28), - an imaging data processing phase implemented by the computing unit (28), comprising: • the segmentation of the imagery data so as to highlight elements belonging to predefined classes on the imagery data, the predefined classes comprising at least one class representative of ground elements (12) and classes representative of other elements of the scene (10), at least one of the highlighted elements being a ground element (12), • the determination of contours, on the image data, corresponding to the intersections between each highlighted ground element (12) and the highlighted elements for other classes, • the extraction of segments, called ground-view segments, representative of each ground element (12) highlighted on the image data, according to the contours determined for said ground element, a thumbnail processing phase implemented by the computing unit (28), comprising: • the segmentation of the thumbnail so as to highlight at least one element on the ground on the thumbnail (12), • the determination of the outlines of each ground element (12) highlighted on the thumbnail, • the extraction of segments, called sky-view segments, representative of each ground element (12) highlighted on the thumbnail, according to the contours determined for said ground element (12), a projection phase, by the computing unit (28), of the segments seen from the ground and the segments seen from the sky in a common coordinate system, the common coordinate system being the coordinate system of the image data or the coordinate system of the thumbnail, an association phase, by the computing unit (28), of the segments seen from the ground and the segments seen from the sky in the common coordinate system, and a determination phase, by the computing unit (28), of at least one position for the optronic system (18) as a function of the association made.

2. A method according to claim 1, wherein the images acquired by the imaging device (20) are panoramic or quasi-panoramic images of the scene (10), the imaging data being a panoramic or quasi-panoramic image or a stream of panoramic or quasi-panoramic images of the scene (10).

3. Method according to claim 1 or 2, wherein the optronic system (18) is in motion in the scene (10), a previous position of the optronic system (18) in the scene (10) being stored in memory (22).

4. A method according to claim 3, wherein the association phase comprises determining a transformation corresponding to the association between segments seen from the ground and segments seen from the sky in the common frame, the phase of determining at least one position for the optronic system (18) comprising determining an approximate position for the optronic system (18) as a function of at least the previous position and the transformation determined.

5. A method according to any one of claims 1 to 4, wherein each ground element (12) belonging to a ground class is a linear element comprising two ends, at the end of the association phase, each end of each ground element (12) considered on the image data is described by two pairs of coordinates, one of the pairs of coordinates being the pixel coordinates of said end on the image data, and the other pair of coordinates being the geographic coordinates of said end on the thumbnail, the phase of determining at least one position for the optronic system (18) comprising the determination of an absolute position and an absolute orientation as a function of the pairs of coordinates determined for each ground element (12).

6. A method according to any one of claims 1 to 5, wherein the association phase comprises the association of segments seen from the ground and segments seen from the sky corresponding to the same directions.

7. A method according to any one of claims 1 to 6, wherein the association determines a transformation such that after application to the elements on the ground, the difference between the segments seen from the ground and the segments seen from the sky is minimal compared to other possible associations.

8. A method according to any one of claims 1 to 7, wherein the ground elements (12) are selected from one or more of the following: - streets, alleys, avenues, or boulevards in urban areas, - highways, national roads, departmental roads in urban, peri-urban or rural areas, - agricultural, forest, mountain roads in rural areas, - roads and tracks in desert areas, and - fields, or open uncultivated areas in rural areas.

9. A method according to any one of claims 1 to 8, wherein the method includes an initialization phase consisting of determining an initial position and an initial orientation of the optronic system (18).

10. A method according to any one of claims 1 to 9, wherein the optronic system (18) is deployed on a platform jointly with a navigation instrument, such as an inertial navigation system or inertial measurement unit, the method comprising a phase of recalibrating the navigation instrument according to a pose determined by the optronic system (18).

11. A method according to any one of claims 1 to 10, wherein the segmentations performed on the image data and on the thumbnail are implemented by a deep learning algorithm, preferably a semantic segmentation algorithm.

12. A method according to any one of claims 1 to 11, wherein the common frame considered during the projection phase is the frame of the image data.

13. A method according to any one of claims 1 to 12, wherein the imaging device (20) comprises several optronic sensors distributed around the vehicle, the imaging data being constructed from images captured by the optronic sensors, said images being dated and assembled using beam-fitting processing to establish the imaging data.

14. An optronic system (18) for determining at least one position by means of an optronic system (18) in a scene (10), the scene (10) comprising ground features (12) with known geographic coordinates, such as roads or paths, the optronic system (18) comprising elements integrated into said optronic system

15. (18) and configured to implement a method according to any one of claims 1 to 13, the integrated elements comprising at least the following elements: - an imaging device (20) suitable for acquiring images of the scene (10) as seen from the optronic system (18), - a memory (22) in which is stored a bird's-eye view representation of an area of ​​space comprising at least the scene (10), with ground elements (12) being geo-referenced to the bird's-eye view representation, and - a calculation unit (28). Vehicle, such as a land vehicle, comprising an optronic system (18) according to claim 14.

Images