Method for automatically realigning the navigation of an optronic system moving around a navigation zone

Abstract

The present invention relates to a method for automatically realigning the navigation of an optronic system (18) moving around a navigation zone, the method comprising, over time, the following steps: estimating, by means of a navigation device, a current position and a current orientation for the optronic system (18); determining reference objects visible from the optronic system (18), referred to as expected objects; acquiring an imaging datum; searching the imaging datum for objects referred to as terrain objects; determining, for each terrain object, an angular deviation on the basis of the terrain objects and the expected objects; and updating and maintaining the navigation of the navigation device on the basis of the angular deviation determined for each terrain object.

Description

[0001] TITLE: Automatic navigation recalibration method for an optronic system operating in a navigation zone

[0002] The present invention relates to a method for automatically recalibrating the navigation of an optronic system operating within a navigation zone. The present invention also relates to such an optronic system and an associated vehicle.

[0003] The field of the invention relates to the autonomous recalibration of vehicle navigation, in particular military vehicles, also addressed under the more general term PNT for position navigation timing in English (translated into French as position navigation temps).

[0004] In particular, the aim is to solve the problem of navigation information drift in a vehicle equipped with an inertial measurement unit (IMU) and an optronic sensor in a contested environment. A contested environment is defined as one potentially without access to GNSS (Global Navigation Satellite System) signals, without means of communication, without an electromagnetic emission sensor (vs. discretion and power consumption), without collaborating elements in the environment (beacons, specific markings, etc.) as in US 2007 / 0276590 A, and without the use of pre-existing ground-based imagery as in US 2012 / 0300020 A. Such imagery is not available or updated in hostile areas (e.g., enemy territory).

[0005] It is desired that the navigation recalibration can be carried out both in structured and unstructured environments, in the broadest sense, on all types of terrain on which the vehicle carrying the optronic system is likely to move on the surface of the planet.

[0006] The state of the art provides solutions for navigation assistance in passenger vehicles and civilian transport, with applications and research dedicated to autonomous driving. These vehicles generally carry appropriate sensors, with the systematic use of a GNSS receiver and the frequent use of active sensors such as LiDAR and / or radar and / or active imaging sensors. The operating environment is often structured (urban). Obstacle avoidance and route planning functions are often emphasized, and off-road driving constraints are of little interest in the civilian sector. These solutions are therefore not suitable for contested environments, particularly military ones where discretion is paramount.

[0007] There is therefore a need for an automatic navigation recalibration solution for an optronic system that can be used in a contested environment, without GNSS signals, without emission and without reception of electromagnetic waves emitted from objects or a third-party network, and without emitting electromagnetic waves (in the radar and optronic domains from ultraviolet to infrared).

[0008] To this end, the invention relates to a method for automatically recalibrating the navigation of an optronic system operating within a navigation zone, the optronic system comprising the following elements: a navigation device capable of estimating over time at least one position and orientation for the optronic system and of guiding the movement of the optronic system according to the estimated position and orientation, an imaging device capable of acquiring images of the environment seen from the optronic system, a memory in which is stored:

[0009] • the positions of reference objects that have been previously identified in the navigation area, each reference object belonging to a class of objects,

[0010] • a database of object classes grouping the object classes of the reference objects, a computing unit, the process being implemented by the optronic system and comprising over time, the following steps: the estimation, by the navigation device, of a current position and a current orientation for the optronic system, the determination, by the computing unit, of reference objects visible from the optronic system, called expected objects, and of the object classes of said expected objects, the determination being carried out as a function of the current position and current orientation estimated for the optronic system, and the memorized positions of the reference objects, each expected object being assumed to be visible by the optronic system along an expected direction specific to said expected object, the expected direction being the direction of the line passing through the memorized position of the expected object and the current position of the optronic system,The acquisition of imaging data by the imaging device, the imaging data being an image or a stream of images of the environment seen from the optronic system; the search on the imaging data, by the computing unit, for objects, called terrain objects, belonging to the object classes of the expected objects; the determination, by the computing unit, of a direction, called terrain direction, for each identified terrain object based on the position of the terrain object on the imaging data; the determination, by the computing unit, for each terrain object, of the angular deviation between the terrain direction and the expected direction for the corresponding expected object; and the updating and maintenance of the navigation of the navigation device based on the angular deviation determined for each terrain object.

[0011] Depending on other advantageous aspects, the process includes one or more of the following characteristics, taken individually or in all technically possible combinations:

[0012] - for each class of objects, the expected objects are the reference objects located inside a circle centered on the current position of the navigation device and with a predetermined radius for said class of objects;

[0013] - a digital terrain model of the navigation area is also stored in the memory of the optronic system, the step of determining the expected objects including the identification, according to the digital terrain model, of the expected objects not visible from the optronic system and the filtering of said identified objects so as to retain only the expected objects visible from the optronic system;

[0014] - The field object search step includes, for each expected object:

[0015] - the extraction of one or more thumbnails from the image data based on the expected direction of the expected object, and

[0016] - the search, in each thumbnail, for a terrain object belonging to the class of objects of the expected object;

[0017] - the process includes a step of setting up a tracking system for each terrain object, the steps of determining angular deviation, and of updating and maintaining the navigation being repeated for each tracked terrain object as long as the tracking of the terrain object is ensured;

[0018] - The search step also includes the identification of at least one predetermined additional object on the imaging data, the additional object not belonging to the object classes of the reference objects, the additional object having an absolute position,

[0019] - The ground direction determination step also includes determining a ground direction for each additional object identified based on the position of the additional object on the image data; - The angular deviation determination step also includes determining the angular deviation between the ground direction and an expected direction for the additional object, the expected direction being the direction of the line passing through the absolute position of the additional object and the current position of the optronic system.

[0020] - the step of updating and maintaining the navigation of the navigation device is also a function of the angular deviation determined for the additional object;

[0021] - the additional object is the sun or a celestial body, the absolute position of the additional object being obtained according to ephemeris files or a catalogue of astrometric data stored in the memory of the optronic system;

[0022] - the navigation update and maintenance step includes comparing the angular deviation of each identified object to the variance corresponding to the difference between the expected direction and the ground direction, the consideration of the angular deviation of said identified object for the update and recalibration of the navigation being a function of the result of the comparison;

[0023] - at the end of the navigation update and maintenance step, at least the corrected position of the optronic system is stored in the optronic system's memory for later reuse.

[0024] This description also relates to an optronic system configured to implement a process as described above, the optronic system comprising the following elements: a navigation device capable of estimating over time at least one position and orientation for the optronic system and of guiding the movement of the optronic system according to the estimated position and orientation, an imaging device capable of acquiring images of the environment seen from the optronic system, a memory in which is stored:

[0025] • the positions of reference objects that have been previously identified in the navigation area, each reference object belonging to a class of objects,

[0026] • a database of object classes grouping the object classes of the reference objects, a unit of calculation.

[0027] This description also applies to a vehicle, such as a land vehicle, comprising an optronic system as described above.

[0028] The invention will become clearer upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which: [Fig. 1] Figure 1 is a schematic representation of an example of a vehicle equipped with an optronic system evolving in a navigation area in which reference objects belonging to object classes have been previously identified,

[0029] [Fig 2] Figure 2 is a schematic representation of an example of an optronic system comprising elements integrated into said system,

[0030] [Fig 3] Figure 3 is a flowchart of an example of the implementation of an automatic navigation recalibration process for an optronic system operating within a navigation zone, and

[0031] [Fig 4] Figure 4 is an example illustrating a step in determining reference objects visible from the optronic system, called expected objects, and the classes of said expected objects.

[0032] In the following description, the absolute (geographic) orientation of an object in a scene is defined as the vector joining the coordinate system to the object; the associated direction is the line containing this vector. This absolute orientation is characterized by angles expressed relative to a geographic reference. The most commonly used are the azimuth angle, which expresses the orientation in a locally horizontal plane (tangent to the ellipsoid associated with the geoid) relative to the local geographic meridian, and the elevation angle (or inclination angle), which expresses the orientation in a vertical plane relative to the locally horizontal plane. A compass is typically used to measure azimuth. An inclinometer is typically used to measure elevation. A relative orientation is defined with respect to another reference orientation (i.e., an angular difference between two orientations), characterized by the bearing angle in the horizontal plane and the elevation angle in the vertical plane.A goniometer typically allows the measurement of a deposit and a site.

[0033] The attitude of an element is defined as the information that allows the element to be fully oriented in a frame of reference covering the 3 dimensions of geographic space (for example, in a minimal way with the 3 Euler angles of roll, pitch and yaw).

[0034] A scene 10 is illustrated as an example in Figure 1. A scene designates a theater of operations, that is, the place where an action takes place. The scene is therefore an extended space with sufficient dimensions to allow the action to unfold. The scene is typically an outdoor space.

[0035] Scene 10 belongs to a navigation zone. A navigation zone is, for example, a region of space extending over several kilometers. The navigation zone is also referred to as an operations zone, mission zone, or scene.

[0036] Scene 10 is, for example, a road network. Alternatively, scene 10 is a portion of space outside the road network. The navigation area includes reference elements, also called landmarks or reference structures, with known geographic coordinates and accuracy.

[0037] Each reference element is a fixed and prominent object or feature within the navigation area. The coordinates (latitude, longitude) of each reference element are known. Optionally, the altitude of each reference element is also known. Preferably, the accuracy of the latitude and longitude coordinates of each reference element is known from the high-quality metadata associated with an orthophoto representing the navigation area as seen from above. Similarly, the altitude accuracy is known from the metadata of a digital terrain model that provides the altitudes of ground points within the navigation area used in the vicinity of the reference element.

[0038] Reference elements are, for example, points belonging to the following: a construction (building, bell tower, lighthouse, road, bridge, etc.) whose coordinates can be found on an ortho-image of satellite or airborne origin or a topographic map and a natural element (mountain, rock, hilltop, vegetation, tree, etc.) whose coordinates can be found on an ortho-image of satellite or airborne origin.

[0039] In the example illustrated by Figure 1, the reference elements are mountains E1, water towers E2, a tree E3 and a road E4 with a crossing E5.

[0040] As illustrated by the example in Figure 1, an optronic system 18 evolves in scene 10.

[0041] The optronic system 18 is preferably mounted on a vehicle 19.

[0042] Vehicle 19 is, for example, a land vehicle as shown in Figure 1. Vehicle 19 is, for example, a military type, such as an assault tank or an infantry fighting vehicle (IFV). Such a military vehicle is specifically designed to carry multiple weapons and to protect the operator(s) inside the vehicle.

[0043] Alternatively, vehicle 19 is an aerial vehicle (aircraft, drone) or a maritime vehicle (boat).

[0044] As illustrated by Figure 2, the optronic system 18 comprises at least the following elements: a navigation device 20, an imaging device 21, a memory 22 and a computing unit 28. Optionally, the optronic system 18 further comprises a visualization element 30.

[0045] Preferably, these elements are integrated into the optronic system 18. By the term "integrated", it is understood that the elements are incorporated physically and / or software into said optronic system 18. Such elements therefore form a single block in the optronic system 18.

[0046] The navigation device 20 is capable of estimating over time at least one position and orientation for the optronic system 18 and of guiding the movement of the optronic system 18 according to the estimated position and orientation.

[0047] The navigation device 20 is, for example, an inertial component, for example an inertial measurement unit.

[0048] The imaging device 21 (also referred to as "sensor" in the description) is designed to acquire images of the environment seen from the optronic system 18, i.e. images of the scene 10.

[0049] Typically, the native images acquired by the imaging device 21 can be resampled to a specific geometry or projection. The images are composed of pixels identified by pixel coordinates. Each pixel corresponds to a direction characterized by its angular values ​​(bearing, elevation) in a frame of reference linked to the optronic sensor (imaging device 21). The sensor mounting on the vehicle is determined, for example, by a calibration procedure, which allows the angular information from a direction in the sensor's frame of reference to be transferred to a vehicle frame of reference. In the case of multiple distributed sensors, the sensor mounting allows the directions recorded by each sensor to be processed in a common vehicle frame of reference.

[0050] Preferably, the images acquired by the imaging device 21 are panoramic or near-panoramic images. An image is considered panoramic when it images a scene over 360° in azimuth. The elevation range is, for example, between 75° and -15°. A near-panoramic image is a panoramic image with occultation zones. An occultation zone is an angular sector in azimuth for which the scene is not imaged. An occultation zone extends, for example, at most 25° in azimuth and over the entire elevation range of the image.

[0051] In practice, elevation ranges can be limited by objects in the image. Image processing can also reduce the areas actually acquired when these areas correspond to regions with low a priori information, such as elevated sites representing the sky, for example.

[0052] Advantageously, the imaging device 21 is capable of operating in several spectral bands, for example, in the visible and in the infrared.

[0053] The imaging device 21 is a passive device. This allows for electromagnetic discretion and limits energy consumption. The imaging device 21 is, for example, composed of a single sensor (camera). This allows for optimal spatial and temporal coherence for the acquired images.

[0054] Alternatively, the imaging device 21 is formed by a set of sensors (cameras) and their images are then time-synchronized and geometrically adjusted in order to extract useful information as if one had an instantaneous view of the scene (where applicable panoramic).

[0055] Preferably, the imaging device 21 is an omnidirectional imaging device, that is to say an imaging device capable of providing 360° images in azimuth and up to 180° in elevation.

[0056] In another variant, the imaging system 21 does not image the entire 360° due to masking or an architecture composed of a set of non-overlapping distributed cameras. As mentioned previously, the images are then described as quasi-panoramic.

[0057] In one example, the imaging device 21 includes one or more MINERVA sensors developed by THALES.

[0058] Memory 22 contains the positions of reference objects that have been previously identified in the navigation area. The position of each reference object is typically an absolute position (latitude, longitude, preferably altitude).

[0059] Each reference object belongs to an object class. An object class is defined as a category of objects.

[0060] Memory 22 also includes in memory a class database of objects grouping the object classes of the reference objects.

[0061] The classes of reference objects typically correspond to objects relevant to address both robustness through their rarity (low density) and precision (object in sufficient number and at best the largest); and this to allow an association using for example the symbolic characteristics of these classes.

[0062] Preferably, object classes correspond to objects with a vertical extension (extending vertically).

[0063] Preferably, object classes allow the navigation area to be described by embedded information in the form of a map element from which the position of objects belonging to specific classes can be extracted (water tower, bell tower, electricity and cell phone repeater tower, electricity and telephone pole, wind turbine, etc.). These objects are not intended to be found specifically (for example: we will not record the specific characteristics of the water tower of a specific village - height, diameter, particular profile - we will simply indicate that in this specific village, there is an object of the generic class "water tower" with its position).

[0064] For areas with low object density, object classes include, for example, the house and tree classes to characterize a dwelling or an isolated tree whose position is unambiguous, because it is alone in a large area, with regard to the information known a priori of the positioning and orientation of the vehicle 19.

[0065] Thus, the choice of classes to use according to the navigation area is subsequently adjusted and / or prioritized according to the existence and / or abundance of classes in the area: for example, the wind class will be chosen in Western Europe and the isolated tree class in semi-arid areas.

[0066] In one implementation example, memory 22 includes a bird's-eye view representation of the navigation area. This bird's-eye view representation is, for example, an orthophoto, such as a satellite image or an aerial photograph. Alternatively, the bird's-eye view representation is a map in the form of a digital image, such as a topographic or symbolic map (of the type produced by the IGN for the French National Geographic Institute). Alternatively, the bird's-eye view representation is a fusion of an image and a map. The reference elements of the navigation area are then georeferenced to the bird's-eye view representation. Preferably, each point (pixel, element) in the bird's-eye view representation is associated with geographic coordinates.

[0067] The geographic coordinates of the reference elements are, for example, expressed as a latitude value, a longitude value, and optionally an altitude value. The accuracy errors associated with these data are preferably also provided.

[0068] Preferably, memory 22 includes ephemerides giving the position of the sun and other bodies of the solar system according to the dating.

[0069] Preferably, memory 22 also includes a geometric image capture model. This image capture model allows a pixel of one of the images from the optronic system 18 to be associated with a spatial direction described by two angles (bearing, elevation) in the coordinate system of the frame attached to the optronic system 18. This frame of the optronic system 18 has an attitude relative to the local geographic coordinate system, and the three (Euler) angles characterizing this attitude constitute three of the six unknowns of the system's pose to be determined. Thus, any point of a reference element of the scene having coordinates in an optronic image has a relative direction in the frame of the optronic system 18. The processing unit 28 is designed to receive data from the other elements of the optronic system 18, including images from the imaging device 21, data stored in memory 22, and data from the navigation device 20.

[0070] Calculation unit 28 is, for example, a processor.

[0071] In one example, the computing unit 28 interacts with a computer program product that includes an information storage medium. The information storage medium is readable by the computing unit 28.

[0072] A readable information medium is a suitable medium for storing electronic instructions and capable of being connected to a computer system bus. Examples of readable information media include optical discs, CD-ROMs, magneto-optical discs, ROMs, RAMs, EPROMs, EEPROMs, magnetic cards, optical cards, and USB flash drives. The computer program product, comprising program instructions, is stored on the information medium.

[0073] The computer program can be loaded onto the computing unit 28 and leads to the implementation of steps of an automatic recalibration process of the navigation of an optronic system 18 evolving in a navigation area, when the computer program is implemented on the computing unit 28 as will be described later in the description.

[0074] The display element 30 is suitable for displaying images from the imaging device 21 and / or data stored in memory 22.

[0075] The display element 30 is, for example, a display, such as an OLED screen.

[0076] The operation of the optronic system 18, which involves the implementation of an automatic navigation recalibration process for an optronic system 18 evolving in a navigation zone, will now be described with reference to the flowchart in Figure 3, and Figure 4, which refers to an example of a step in the process.

[0077] The recalibration process is implemented by the optronic system 18.

[0078] The steps of the process are implemented over time (real time). Preferably, the different steps of the process are implemented continuously, for example at the acquisition frequency of the imaging device 20.

[0079] At initialization, the position and orientation of the optronic system 18 are provided to the navigation device 20. For example, the initial position and orientation are given either by a GNSS (if it is reliable at the mission start location), or by bearings from georeferenced landmarks, or by any other static positioning method such as a bearing. ESTIMATION OF A CURRENT POSING

[0080] The process includes a step 90 of estimation, by the navigation device 20, of a current position and a current orientation for the optronic system 18. These estimations are subject to errors depending on the drift of the navigation device 20. It is this drift that we seek to correct in the following steps of the process.

[0081] DETERMINATION OF EXPECTED OBJECTS AND OBJECT CLASSES

[0082] The process includes a step 100 of determining, by the computing unit 28, reference objects visible from the optronic system 18, called expected objects, and classes of objects of said expected objects.

[0083] The determination is made based on the current position and current orientation estimated for the optronic system 18, and the stored positions of the reference objects.

[0084] Each expected object is assumed to be visible to the optronic system 18 in the vicinity of an expected direction specific to that object. The expected direction is the direction of the line passing through the stored position of the expected object and the current position of the optronic system 18.

[0085] Preferably, the precision associated with the expected objects is also determined. The precision is, for example, a precision stored in memory 22 for each reference object.

[0086] In one example implementation, for each class of objects, the expected objects are the reference objects located inside a circle centered on the current position of the navigation device 20 and with a predetermined radius for that class of objects. The classes of expected objects are then retrieved from memory once the expected objects have been identified.

[0087] Thus, the expected objects are extracted from geographic products around the estimated position of vehicle 19 within a radius depending on the resolution of the optronic sensor and the class of the object (thus if for example the optronic sensor has a resolution of 1 milliradian, it would be able to see a water tower 10 meters in diameter at 10 km, and a pylon 1 meter in diameter at 1 km).

[0088] Figure 4 illustrates an example of determining expected objects based on the last estimated position P of the optronic system 18. The circle with the smallest radius, denoted C1, is specific to the class of reference objects represented by triangles. The circle with the intermediate radius, denoted C2, is specific to the class of reference objects represented by circles. The circle with the largest radius, denoted C3, is specific to the class of reference objects represented by stars. As can be seen in this figure, the extracted reference object classes are triangles, circles, and stars. For the triangle class, only the position of the triangle within circle C1 is retained. For the circle class, only the position of the circle within circle C2 is retained. For the star class, the position of each of the two stars shown in Figure 4 is retained (since both are within circle C3).

[0089] Note that in Figure 4, the directions of the various expected objects are very disparate and do not present any ambiguity in the correspondence between them. In the case where two objects are close in direction (based on the precision of their directions), the correct association can be achieved by the difference in category between the expected objects, and for objects of the same class, the distance that affects their size can also be used.

[0090] Preferably, a digital terrain model (DTM) of the navigation area is also stored in memory 22 of the optronic system 18. The determination step 100 includes the identification, based on the digital terrain model, of expected objects not visible from the optronic system 18 and the filtering of said identified objects so as to retain only the expected objects visible from the optronic system 18. This makes it possible to eliminate objects whose intervisibility with the vehicle 19 is not assured.

[0091] In a specific (optional) embodiment, for areas with low object density, characteristic variations in the terrain can be used, if the navigation area is suitable. This could be achieved, for example, with a digital terrain model (DTM) (mountain summit with known altitude) and also the profile of a mountain visible from vehicle 19, using specific artificial intelligence training (e.g., extraction of the land / sky boundary through semantic segmentation and definition of the mountain profile). An analysis of the terrain slopes from the DTM allows for the predetermination of areas suitable for creating natural landmarks with visible elevation changes, for use in planimetric registration.

[0092] In one implementation example, the position is determined from the pixel coordinates of objects on a bird's-eye view representation (geographic product) and the metadata associated with this bird's-eye view representation. Their accuracy depends on the quality of the metadata (generalization error, bias error: these errors being associated with the geographic product) and the extraction process (parallax error, error in determining the object's center, etc.).

[0093] IMAGING DATA ACQUISITION The process includes a step 110 of acquiring imaging data by the imaging device 21.

[0094] Image data is an image or image stream of the environment as seen from the optronic system 18.

[0095] Preferably, the image data is a panoramic (or quasi-panoramic) image, or a stream of panoramic (or quasi-panoramic) images of scene 10.

[0096] Image acquisition can take place with the aircraft stationary or in motion. The speed of movement must be compatible with the integration time of the imaging device to avoid blurring the image. For example, a speed of approximately 50 km / h can be achieved with an integration time of a few milliseconds. Preferably, the position maintained by the inertial measurement unit (IMU) should be acquired simultaneously with the image acquisition.

[0097] In one example, the imagery data is constructed from videos of optronic sensors distributed around vehicle 19. The images from said sensors are time-stamped and assembled using beam-fitting processing to establish a panoramic image or a stream of panoramic images of scene 10.

[0098] Knowledge of the image assembly, describing their position and attitude within a common vehicle reference frame, can be refined as needed to meet performance objectives. This is achieved by favoring the use of observations comprised of: matching image primitives in overlapping image areas; to improve accuracy, small point primitives with sub-pixel precision are used.

[0099] Matching objects with known ground coordinates to their corresponding image coordinates;

[0100] A mixture of the two previous types of observations: relative and absolute.

[0101] OBJECT SEARCH IN THE FIELD

[0102] The process includes a step 120 of searching, by the computing unit 28, for objects, called terrain objects, belonging to the classes of objects of the expected objects.

[0103] In an example implementation, search step 120 involves, for each expected object, extracting one or more thumbnails from the imagery data based on the expected direction for that object. The thumbnails are small compared to the imagery data. For example, at least one thumbnail is centered (or nearly centered) on the expected direction of an expected object. In this example, search step 120 involves searching, within each thumbnail, for a terrain object belonging to the same object class as the expected object from which the thumbnail was extracted.

[0104] The search is, for example, implemented by an artificial intelligence algorithm, such as a classifier. The classifier is, for example, implemented by a neural network.

[0105] Each identified terrain object has a direction which is obtained as a function of the pixel coordinates of the terrain object extracted from the panoramic data since each pixel corresponds to a direction characterized by its angular values ​​(bearing, site).

[0106] In one implementation example, if at least two objects are identified, a check is performed to verify that selected objects exist within that angular sector and that the angular separation in the image is consistent with the expected separation between these two objects. In this case, each object in the thumbnail can be unambiguously associated with the reference objects.

[0107] In one example implementation, image data is presented in a panoramic view on a display, with the option to overlay expected object classes (thumbnails) onto a particular image. It is also possible to display an orthographic or topographic view with the option to overlay:

[0108] The set of object classes on a predefined region.

[0109] Certain classes of particular objects or in a small neighborhood around the vehicle 19.

[0110] Candidate objects in terms of their potential presence on a typical optronic image.

[0111] The history of the trajectory of vehicle 19.

[0112] DETERMINATION OF FIELD DIRECTION

[0113] The process includes a step 130 of determining, by the computing unit 28, a direction, called ground direction, for each ground object identified as a function of the position of the ground object on the image data (coordinates) and the position and orientation of the sensor (optronic system) presumed to be corrected.

[0114] In an example implementation, the position of each terrain object corresponds to at least one pixel in the image data. Each pixel is characterized by an angular value (site, bearing), and the terrain direction of the terrain object is then obtained based on the associated angular value of that pixel. The geographic direction is obtained from the sensor coordinate system direction, the sensor mounting, and the vehicle positioning. ANGULAR DEVIATION DETERMINATION

[0115] The process includes a step 140 of determining, by the calculation unit 28, for each terrain object, the angular deviation between the terrain direction and the expected direction for the expected object corresponding to said terrain object.

[0116] In particular, the reference object corresponding to said terrain object is the object in the thumbnail that was used to obtain the terrain object (expected direction of the terrain object). The difference between the expected direction of the object and its direction corresponding to its position in the sensor image is all the greater the further the approximate pose of the vehicle deviates from the truth.

[0117] NAVIGATION UPDATE AND MAINTENANCE

[0118] The process includes a step 150 of updating and maintaining the navigation of the navigation device 20 based on the angular deviation determined for each terrain object.

[0119] Navigation is maintained automatically, i.e. without requiring intervention from the crew or occupants of vehicle 19.

[0120] The update involves, for example, correcting the position and orientation of the optronic device 18 by an angular deviation that is a function of the angular deviations determined for each terrain object. For example, the correction consists of a weighted average of the angular deviations determined for each terrain object.

[0121] The update technique can be adapted to the number of object classes associated in the image and the number of parameters actually estimated. This number determines the system's degrees of freedom and the minimum number of observations required to perform a single navigation update (batch estimation). This number of parameters can be limited to 2 if the analysis is restricted to the horizontal plane and the vehicle's orientation is assumed to be of sufficient quality. However, for poor orientation, the position state vector increases to 3 parameters, including the orientation that determines the vehicle's heading. It can be set at 6 if the goal is to estimate the vehicle's spatial position and attitude using these three Euler angles. It can also take an intermediate value if the spatial position and heading are estimated without attempting to estimate the vehicle's attitude. In this case, roll and pitch measurements from the inertial measurement unit (IMU) are used, for example.Each observation direction yields two observation equations. Depending on the size of the state vector to be estimated and the number of observations available in an image, one of the following is used: a batch procedure estimating all parameters using the set of observations via a least-squares approach; or a sequential procedure that integrates measurements one by one over time into an estimation filter of the EKF, UKF, or particle type, in their most common applications. The variances of the observations are used to initialize the filter's measurement matrix, and the approximate vehicle pose is used to constrain the estimated state. Preferably, step 150, the navigation update and maintenance, includes comparing the angular deviation of each terrain object to the variance corresponding to the difference between the expected direction and the actual terrain direction.The consideration of the angular deviation of said terrain object for the updating and recalibration of navigation depends on the result of the comparison.

[0122] In an optional implementation example, the search step also includes identifying at least one predetermined additional object in the image data. This additional object is excluded from the object classes of the reference objects, and since the additional object has an absolute position, the ground direction determination step also includes determining a ground direction for each identified additional object, based on the object's position in the image data. In one implementation example, the position of each additional object corresponds to at least one pixel in the image data. Since each pixel is characterized by an angular value (site, bearing), the ground direction of the additional object is then obtained based on the associated angular value of that pixel.The angular deviation determination step also includes determining the angular deviation between the ground direction and an expected direction for the additional object, the expected direction being the direction of the line passing through the absolute position of the additional object and the current position of the optronic system 18, the step of updating and maintaining the navigation of the navigation device 20 also being a function of the angular deviation determined for the additional object.

[0123] For example, the additional object is the sun or a celestial body. The absolute position of the additional object is obtained, for example, from ephemeris files or an astrometric data catalog stored in memory 22 of the optronic system 18. In an example implementation, if 3 or more objects of the expected classes have been found and classified in the image data, then a planar position and bearing can be calculated using the method described in patent application FR 3 142 247. Knowledge of the sensor mounting allows the azimuth to be determined. If the DEM is used as an option, the planar position can be enriched with an altitude (terrain + sensor height).

[0124] This involves estimating a 6D pose as described in patent application FR 3 142 247 (Bayesian estimation method)

[0125] And in all cases, we calculate the errors associated with these quantities.

[0126] If two objects of the expected classes are detected in the image, the planar orientation (yaw) measured by the IMU and maintained by the filter is used, for example. The planar position of vehicle 19 is then calculated using the image observations and the ground coordinates of the two objects of the expected classes, employing a simplified bearing-type approach in which the bearing (zero reading of the goniometer) is fixed according to the sensor setup and the azimuth value predicted by the filter at the image date. This method also allows for the calculation of the accuracy of the position obtained from the accuracy of the ground and image coordinates, as well as the yaw accuracy.

[0127] To ensure the robustness of the IMU yaw value, use some measurements collected in the past or in the filter's state history to assess whether the trajectory is locally rectilinear. This is because it has been observed that IMU yaw errors occur primarily during turns. Therefore, measurements taken during turns should be rejected to avoid feeding the filter with erroneous observations due to significant azimuth errors.

[0128] To determine the altitude of vehicle 19 by adding its height to the altitude value interpolated in the DEM based on the planar position in order to obtain a 3D position and its accuracy

[0129] Send to the filter the position obtained as observations as well as the 2 directional observations determined on the 2 objects of the expected classes.

[0130] If a single object of the expected classes is detected in the image, then we limit ourselves to determining the direction of vehicle 19 and its accuracy from this observation to infer the navigation filter.

[0131] Preferably, at the end of the navigation update and maintenance phase, at least the corrected position of the optronic system 18 is stored in the memory 22 of the optronic system 18 for later reuse, for example, in the event of an interruption of the navigation update and maintenance phases. This can also be used for debriefing, evaluation, or fine-tuning phases during or after the mission.

[0132] IMPLEMENTATION OF A MONITORING SYSTEM

[0133] Optionally, the process includes a step 160 for setting up tracking for each terrain object. The steps for determining angular deviation 140 and for updating and maintaining navigation 150 are then repeated for each tracked terrain object as long as tracking of the terrain object is maintained (as long as the object is identified in the imagery data). Object loss occurs, for example, due to moving away or being obscured.

[0134] In one implementation example, this tracking capability produces an observation in the form of a series of time-domain directions (Azimuth / elevation) and their accuracy on the class of an object with the same ground coordinates.

[0135] In the favorable case, where vehicle 19 is verified to be traveling in a near-straight line at a uniform speed, i.e., Uniform Rectilinear Motion (URM), tracking an object allows it to be viewed from different angles and the four unknowns of position and velocity characterizing URM in the horizontal plane to be easily determined. The URM conditions are verified using information from the navigation filter (continuously updated by the IMU). In this case, a summarized navigation state (tracklet) is estimated in the form of position and velocity in the local plane. The local plane is assumed to be horizontal or, ideally, determined from the DEM when available as a first-degree polynomial in local Cartesian coordinates; the coefficients of the polynomial are estimated from the altitudes of the nodes near the position of vehicle 19.

[0136] By way of example, object tracking within the image is performed using a tracking device (based on DeepSort-type artificial intelligence tracking, or on conventional computer vision tracking: using object contrast and / or correlation between successive images...). The inertial measurement unit (IMU) will enable inertial tracking of the object in case of momentary obscuration. In case of prolonged obscuration, the sector will be monitored for that object in the expected area.

[0137] In one implementation example, if multiple objects are extracted from the image, they are tracked by a tracking function. This capability produces observations in the form of a series of time-varying directions (azimuth / elevation) derived from their coordinates in the images. Each series corresponds to a unique object class with known ground coordinates and accuracies. The series can be identical or distinct. The image and ground coordinates of objects within the same image are distinct.

[0138] The observation series allows us to estimate an intermediate pose state and its variance (tracklet) as before, but by placing fewer constraints on the trajectory of vehicle 19.

[0139] ADVANTAGES AND CONCLUSION

[0140] The invention solves the problem of recalibrating the navigation of a vehicle, particularly a military one, without GNSS, by using inertial measurements exhibiting drift and classes of objects extracted from existing data (geographic products viewed from the sky) which have reference ground coordinates and from optronic images (view from the ground).

[0141] Using object classes rather than traditional landmarks allows for the association of more generic symbolic information than the geometric or radiometric attributes associated with traditional landmarks.

[0142] More specifically, focusing on objects of interest as a generic class eliminates the need for precise characterization to identify them as elements of their class (unlike specialized military personnel who use "landmark notebooks" precisely describing the landmarks to be found in a given scene). These objects can thus be located at great distances from the sensor with very low resolution: a point in the expected direction (without other points in close angular proximity) will indicate that the object is indeed present. This solution will allow for the exploitation of features found in abundance in rural areas: a cluster of pylons on a power line, a wind farm, etc.

[0143] To illustrate the process, it will preferably exploit the angular arrangement between 3 objects belonging to determined classes, rather than the specific recognition of each object according to its specific characteristics (dimension, texture, profile or shape of the object).

[0144] The process is compatible for updating and maintaining navigation in structured environments, but also unstructured ones; that is to say, including outside built-up areas and outside of paved roads such as a path or a field, as long as the scene includes a minimum of observable objects of interest.

[0145] Those skilled in the art will understand that the order of the different phases is given by way of example. For instance, the determination step 100 can be reversed with the acquisition step 110. Those skilled in the art will understand that the embodiments described above can be combined to form new embodiments provided they are technically compatible.

Claims

DEMANDS 1. Method for automatically recalibrating the navigation of an optronic system (18) moving within a navigation zone, the optronic system (18) comprising the following elements: a navigation device (20) capable of estimating over time at least one position and orientation for the optronic system (18) and of guiding the movement of the optronic system (18) according to the estimated position and orientation, an imaging device (21) capable of acquiring images of the environment seen from the optronic system (18), a memory (22) in which is stored: • the positions of reference objects that have been previously identified in the navigation area, each reference object belonging to a class of objects, • a database of object classes grouping the object classes of the reference objects, a computing unit (28), the process being implemented by the optronic system (18) and comprising, over time, the following steps: the estimation, by the navigation device (20), of a current position and a current orientation for the optronic system (18), the determination, by the computing unit (28), of reference objects visible from the optronic system (18), called expected objects, and of the object classes of said expected objects, the determination being carried out as a function of the current position and current orientation estimated for the optronic system (18), and the stored positions of the reference objects, each expected object being assumed to be visible by the optronic system (18) according to an expected direction specific to said expected object,the expected direction being the direction of the line passing through the memorized position of the expected object and the current position of the optronic system (18), the acquisition of an image data by the imaging device (21), the image data being an image or a stream of images of the environment seen from the optronic system (18), the search on the image data, by the computing unit (28), for objects, called terrain objects, belonging to the object classes of the expected objects, the determination, by the computing unit (28), of a direction, called terrain direction, for each terrain object identified as a function of the position of the terrain object on the image data, the determination, by the computing unit (28), for each terrain object, of the angular deviation between the terrain direction and the expected direction for the corresponding expected object of said terrain object, and the updating and maintenance of the navigation of the navigation device (20) according to the angular deviation determined for each terrain object.

2. Method according to claim 1, wherein, for each class of objects, the expected objects are the reference objects located inside a circle centered on the current position of the navigation device (20) and of predetermined radius for said class of objects.

3. Method according to claim 2, wherein a digital terrain model of the navigation area is also stored in the memory (22) of the optronic system (18), the step of determining the expected objects comprising the identification, according to the digital terrain model, of the expected objects not visible from the optronic system (18) and the filtering of said identified objects so as to retain only the expected objects visible from the optronic system (18).

4. A method according to any one of claims 1 to 3, wherein the ground object search step comprises, for each expected object: the extraction of one or more thumbnails from the image data according to the expected direction of the expected object, and the search, in each thumbnail, for a ground object belonging to the class of objects of the expected object.

5. A method according to any one of claims 1 to 4, wherein the method comprises a step of setting up tracking for each terrain object, the steps of determining angular deviation, and of updating and maintaining navigation being repeated for each tracked terrain object as long as tracking of the terrain object is ensured.

6. A method according to any one of claims 1 to 5, wherein: the search step also includes the identification of at least one predetermined additional object on the image data, the additional object not belonging to the object classes of the reference objects, the additional object having an absolute position, The ground direction determination step also includes determining a ground direction for each additional object identified based on the position of the additional object on the image data; the angular deviation determination step also includes determining the angular deviation between the ground direction and an expected direction for the additional object, the expected direction being the direction of the line passing through the absolute position of the additional object and the current position of the optronic system (18); the navigation update and maintenance step of the navigation device (20) is also a function of the angular deviation determined for the additional object.

7. Method according to claim 6, wherein the additional object is the sun or a celestial body, the absolute position of the additional object being obtained as a function of ephemeris files or an astrometric data catalogue stored in the memory (22) of the optronic system (18).

8. A method according to any one of claims 1 to 7, wherein the navigation update and maintenance step includes comparing the angular deviation of each identified object to the variance corresponding to the difference between the expected direction and the ground direction, the consideration of the angular deviation of said identified object for the update and recalibration of the navigation being a function of the result of the comparison.

9. A method according to any one of claims 1 to 8, wherein at the end of the navigation update and maintenance step, at least the corrected position of the optronic system (18) is stored in the memory (22) of the optronic system (18) for later reuse.

10. Optronic system (18) configured to implement a method according to any one of claims 1 to 8, the optronic system (18) comprising the following elements: a navigation device (20) adapted to estimate over time at least one position and orientation for the optronic system (18) and to guide the movement of the optronic system (18) according to the estimated position and orientation, an imaging device (21) adapted to acquire images of the environment seen from the optronic system (18), a memory (22) in which is stored: • the positions of reference objects that have been previously identified in the navigation area, each reference object belonging to a class of objects, • a base of object classes grouping the object classes of the reference objects, a calculation unit (28).

11. Vehicle (19), such as a land vehicle, comprising an optronic system (18) according to claim 9.

Images