Method for automatically recalibrating the navigation of an optronic system operating within a navigation zone

Abstract

Method for Automatic Navigation Recalibration of an Optronic System Operating in a Navigation Zone. The present invention relates to a method for automatically recalibrating the navigation of an optronic system (18) operating in a navigation zone. The method comprises, over time, the following steps: the estimation, by a navigation device, of a current position and orientation for the optronic system (18); the acquisition of image data; the processing of the image data; the processing step comprising segmenting the image data to highlight characteristic elements in the image data; the determination of a set of recalibration corrections for the navigation device; and the updating and maintenance of the navigation of the navigation device based on the determined set of corrections. Figure for the abstract: 1

Description

Title of the invention: Method for automatically recalibrating the navigation of an optronic system operating in a navigation zone

[0001] The present invention relates to a method for automatically recalibrating the navigation of an optronic system operating within a navigation zone. The present invention also relates to such an optronic system and an associated vehicle.

[0002] The field of the invention relates to the autonomous recalibration of vehicle navigation, in particular military vehicles, also referred to under the more general term PNT for position navigation timing in English (translated into French as position navigation temps).

[0003] In particular, the aim is to solve the problem of navigation information drift in a vehicle equipped with an inertial measurement unit (IMU) and an optronic sensor in a contested environment. A contested environment is understood to be one potentially without access to GNSS (Global Navigation Satellite System) signals, without means of communication, without an electromagnetic emission sensor (vs. discretion and power consumption), without collaborating elements in the environment (beacons, specific markings, etc.) as in US 2007 / 0276590 A, and without using pre-existing images taken from the ground as in US 2012 / 0300020 A. Such images are indeed not available or updated in hostile areas (for example, in enemy territory).

[0004] It is desired that the navigation recalibration can be carried out both in structured and unstructured environments, in the broadest sense, on all types of terrain on which the vehicle carrying the optronic system is likely to move on the surface of the planet.

[0005] The prior art provides solutions for navigation assistance in passenger vehicles or civil transport vehicles, with applications and research dedicated to autonomous driving. Vehicles are generally equipped with appropriate sensors, with the systematic use of a GNSS receiver and the frequent use of active sensors such as LiDAR and / or radar and / or active imaging sensors. The operating environment is often structured (urban). Obstacle avoidance and route-finding functions are often emphasized, and off-road driving constraints are of little interest in the civil domain.

[0006] These solutions are therefore not suitable in the context of a contested environment, particularly a military one where the aim is to be as discreet as possible.

[0007] There is therefore a need for an automatic navigation recalibration solution for an optronic system that can be used in a contested environment, without GNSS signals, without emission and without reception of electromagnetic waves emitted from objects or a third-party network, and without emitting electromagnetic waves (in the radar and optronic domains from ultraviolet to infrared).

[0008] To this end, the present description relates to a method for automatically recalibrating the navigation of an optronic system operating in a navigation zone, the navigation zone comprising characteristic elements, the characteristic elements comprising channels, the optronic system comprising the following elements integrated into the optronic system: - a navigation device capable of estimating over time at least one position and orientation for the optronic system and of guiding the movement of the optronic system according to the estimated position and orientation, - an imaging device capable of acquiring images of the environment as seen from the optronic system, - a memory in which a graph of the characteristic elements of the navigation area as seen from the sky is stored, - a unit of calculation,

[0009] the process being implemented by the optronic system and comprising over time the following steps: - the estimation, by the navigation device, of a current position and orientation for the optronic system, - the acquisition of imaging data by the imaging device, the imaging data being an image or a stream of images of the environment seen from the optronic system, - the processing of the imaging data by the computing unit, the processing step including the segmentation of the imaging data in order to highlight characteristic elements, called identified elements, on the imaging data, - the determination, by the computing unit, of a set of corrections for realigning the navigation system, the determination step comprising: • the matching of at least one identified element in the imaging data with a characteristic element of the graph as a function of the current position and orientation of the optronic system, • the determination of a set of corrections to the current position and orientation of the optronic system, to realign at least one identified element in the imaging data with the corresponding characteristic element of the graph, and - the updating and maintenance of the navigation system according to the set of corrections determined.

[0010] According to other advantageous aspects, the process comprises one or more of the following features, taken individually or in all technically possible combinations:

[0011] - during the step of determining a set of corrections, it is determined at less a transverse realignment correction of the current position and orientation of the optronic system relative to the identified element under consideration;

[0012] - the characteristic elements of the graph include road crossings and / or channels with specific curvatures, during the step of determining a set of corrections, when at least one identified element is a crossing of channels and / or a channel with a specific curvature, the matching is also carried out according to the shape of the identified element, and the set of corrections includes at least one longitudinal recalibration correction of the position and current orientation of the optronic system with respect to the identified element considered;

[0013] - during the step of determining a set of corrections, the implementation correspondence includes the search for characteristic elements of the graph, called candidate elements, within a predetermined perimeter around the current position, the element associated with the identified element being chosen from among the candidate elements according to a selection criterion;

[0014] - the selection criterion stipulates that: - the associated element is the candidate element closest to the current position, or - in the presence of several candidate elements at a distance less than a predetermined distance from the current position, the associated element is the candidate element generating a lower association cost when implementing a probabilistic approach, preferably the probabilistic approach implementing a calculation of Mahalanobis distances between each candidate element and the current position;

[0015] - the graph in memory was previously obtained by processing one or more ortho-images of the navigation area so as to highlight the characteristic elements of the navigation area on the ortho-image(s) and to extract the shape of said characteristic elements to form the graph;

[0016] - the characteristic elements are highlighted on the ortho-image(s) by a deep learning algorithm, for example an algorithm implemented by a neural network;

[0017] - a digital terrain model of the navigation area is also stored in the memory of the optronic system, during the step of determining a set of corrections, the matching also being a function of the digital terrain model;

[0018] - in the absence of identified elements on the imaging data, the steps of determination of a set of recalibration corrections, and updating and maintenance of navigation are interrupted, said steps being resumed as soon as at least one element is identified on the image data;

[0019] - at the end of the navigation update and maintenance step, at least the The corrected position of the optronic system is stored in the optronic system's memory to be reused later in case of a possible interruption of the steps of determining a set of recalibration corrections, and updating and maintaining the navigation.

[0020] This description further relates to an optronic system operating in a navigation zone, the navigation zone comprising characteristic elements, the characteristic elements comprising channels, the optronic system being configured to implement a method as described above, the optronic system comprising the following elements: - a navigation device capable of estimating over time at least one position and orientation for the optronic system and of guiding the movement of the optronic system according to the estimated position and orientation, - an imaging device capable of acquiring images of the environment as seen from the optronic system, - a memory in which a graph of the characteristic elements of the navigation area as seen from the sky is stored, and - a unit of calculation.

[0021] This description also relates to a vehicle such as a land vehicle, comprising an optronic system as described above.

[0022] The invention will become clearer upon reading the following description, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0023] [Fig-1] [Fig.1] is a schematic representation of an example of a vehicle equipped with an optronic system operating within a navigation zone comprising characteristic elements,

[0024] [Fig.2] [Fig.2] is a schematic representation of an example of an optronic system comprising elements integrated into said system,

[0025] [Fig.3] [Fig.3] is an example of a portion of a navigation area graph,

[0026] [Fig.4] [Fig.4] is a flowchart of an example of the implementation of a process automatic recalibration of the navigation of an optronic system operating within a navigation zone, and

[0027] [Fig.5] [Fig.5] is a schematic representation of an example illustrating a processing carried out on an image data so as to highlight characteristic elements (routes) on the image data.

[0028] In the following description, the absolute (geographic) orientation of an object in a scene is defined as the vector joining the system to the object; the associated direction being the line containing this vector. This absolute orientation is characterized by angles expressed with respect to a geographic reference. The most commonly used are the azimuth angle, which expresses the orientation in a locally horizontal plane (tangent to the ellipsoid associated with the geoid) with respect to the local geographic meridian, and the elevation angle (or inclination angle), which expresses the orientation in a vertical plane with respect to the locally horizontal plane. A compass is typically used to measure azimuth. An inclinometer is typically used to measure elevation.A relative orientation is defined with respect to another reference orientation (i.e., an angular difference between two orientations), characterized by the bearing angle in the horizontal plane and the elevation angle in the vertical plane. A goniometer typically allows the measurement of bearing and elevation.

[0029] The attitude of an element is defined as the information that allows the element to be fully oriented in a frame of reference covering the 3 dimensions of geographic space (for example, in a minimal way with the 3 Euler angles of roll, pitch and yaw).

[0030] A scene 10 is illustrated by way of example in [Fig. 1]. A scene designates a theater of operations, that is to say, the place where an action takes place. The scene is therefore an extended space with sufficient dimensions to allow the unfolding of an action. The scene is typically an outdoor space.

[0031] Scene 10 belongs to a navigation zone. The navigation zone is, for example, a region of space extending over several kilometers. The navigation zone is also referred to as the operations zone, mission zone, or scene.

[0032] Scene 10 is, for example, a road network. Alternatively, scene 10 is a portion of space outside the road network.

[0033] The navigation area includes characteristic elements having known geographical coordinates (latitude, longitude) and known precision.

[0034] The characteristic elements are ground-level elements. More specifically, ground-level elements include roads, particularly traffic lanes. The characteristic elements thus form a road network. Examples of characteristic elements include roads, highways, paths, or trails.

[0035] The feature elements have known geographic coordinates (latitude, longitude). Optionally, the altitude of each feature element is also known. Preferably, the accuracy of the longitude and latitude coordinates of each feature element is known from the quality metadata associated with an orthophoto representing the navigation area as seen from the sky. Similarly, the altitude accuracy is known from the metadata of a digital terrain model indicating the altitudes at ground points in the navigation area used in the vicinity of the feature element.

[0036] In the example illustrated by [Fig.1], the characteristic elements are a first straight track VI, a second track V2 with a non-zero curvature, and a junction C formed by the intersection of the first track VI and the second track V2.

[0037] As illustrated by the example in [Fig.1], an optronic system 18 evolves in the scene 10.

[0038] The optronic system 18 is preferably mounted on a vehicle 19.

[0039] Vehicle 19 is, for example, a land vehicle as shown in [Fig. 1]. Vehicle 19 is, for example, of a military type, such as an assault tank or an infantry fighting vehicle (IFV). Such a military vehicle is particularly suited to carry a plurality of weapons and to protect the operator(s) inside the vehicle.

[0040] Alternatively, vehicle 19 is an aerial vehicle (aircraft, drone) or a maritime vehicle (boat).

[0041] As illustrated by [Fig.2], the optronic system 18 comprises at least the following elements: a navigation device 20, an imaging device 21, a memory 22 and a computing unit 28. Optionally, the optronic system 18 further comprises a visualization element 30.

[0042] Preferably, these elements are integrated into the optronic system 18. By the term "integrated", it is understood that the elements are incorporated physically and / or software-wise into said optronic system 18. Such elements therefore form a single block in the optronic system 18.

[0043] The navigation device 20 is suitable for estimating over time at least one position and orientation for the optronic system 18 and for guiding the movement of the optronic system 18 according to the estimated position and orientation.

[0044] The navigation device 20 is, for example, an inertial component, for example an inertial measurement unit.

[0045] The imaging device 21 (also referred to as "sensor" in the description) is suitable for acquiring images of the environment seen from the optronic system 18, i.e. images of the scene 10.

[0046] Typically, the native images acquired by the imaging device 21 can be resampled in a specific geometry or projection. The images are composed of pixels identified by pixel coordinates. Each pixel corresponds to a direction characterized by its angular values ​​(bearing, elevation) in a frame of reference linked to the optronic sensor (imaging device 21). The sensor mounting on the vehicle is, for example, determined by a calibration procedure, which allows the angular information from a direction in the sensor frame of reference to be transferred to a vehicle frame of reference. In the case of several distributed sensors, the sensor mounting allows the directions recorded by each sensor to be processed in a common vehicle frame of reference.

[0047] Preferably, the images acquired by the imaging device 21 are panoramic or quasi-panoramic images. An image is said to be panoramic when it images a scene over 360° in azimuth. The elevation range is, for example, between 75° and -15°. A quasi-panoramic image is a panoramic image exhibiting occultation zones. An occultation zone is an angular sector in azimuth for which the scene is not imaged. An occultation zone extends, for example, at most 25° in azimuth and over the entire elevation range of the image.

[0048] In practice, the elevation ranges may be limited by objects in the image. Image processing may also reduce the areas actually acquired when these correspond to areas of low a priori information, such as high sites corresponding to the sky, for example.

[0049] Advantageously, the imaging device 21 is suitable for operating in several spectral bands, for example, in the visible and in the infrared.

[0050] The imaging device 21 is a passive device. This allows for electromagnetic discretion and limits energy consumption.

[0051] The imaging device 21 is, for example, formed by a single sensor (camera). This allows for optimal spatial and temporal coherence for the acquired images.

[0052] Alternatively, the imaging device 21 is formed by a set of sensors (cameras) and their images are then time-stamped and geometrically adjusted in order to extract useful information as if one had an instantaneous view of the scene (where applicable panoramic).

[0053] Preferably, the imaging device 21 is an omnidirectional imaging device, that is to say an imaging device capable of providing images over 360° in azimuth and up to 180° in elevation.

[0054] In another variant, the imaging device 21 does not image the entire 360° due to masking or an architecture composed of a set of non-overlapping distributed cameras. As mentioned previously, the images are then described as quasi-panoramic.

[0055] In another example, the imaging device 21 includes one or more MINERVA sensors developed by THALES.

[0056] Memory 22 includes in memory a graph G of the characteristic elements of the navigation area in view of the sky.

[0057] The graph G can be extracted prior to the mission by a third-party system, or even by processing embedded in the sensor using satellite images or drones capable of acquiring and georeferencing images in view of the sky.

[0058] Alternatively, the graph G can be constructed during the mission (in real time or delayed) from other elements, for example an orthoimage.

[0059] Graph G was not obtained from a 3D model acquired from the ground and requiring prior acquisition by land means (vehicle or pedestrian).

[0060] In one example of implementation, the graph G represents a skeletonization of the characteristic elements of the navigation area, that is to say that only the contours (or the shape) of the characteristic elements are represented on the graph G. The graph G thus represents a road network (made up of roads, highways, paths (rural and forest paths and tracks in desert environments), or trails).

[0061] An example of graph G is illustrated by [Fig.3].

[0062] Preferably, as illustrated in [Fig. 3], the images used to construct the graph G are preprocessed to add geometric information to the graph G (lane width), intersection radii, sections with large radii of curvature, sections with steep slopes, etc. The coordinates of the vertices of the graph G allow for the pre-calculation of length and planar and vertical orientation information. The topology of the vertices is linked to more than two typical edges of L-shaped curve junctions, T-shaped junctions with three branches, X-shaped junctions with four branches, and more. For example, for a junction with a crossing, the relative angles between the different branches are calculated and constitute a strong signature of the vertex, especially when a long path is required to find a vertex with the same crossing topology.

[0063] Thus, in one example, the graph G includes in particular one or more of the following pieces of information: - geometric properties including: • The lengths and widths of the roads, as well as their planar orientations, can be deduced from the positions of the vertices in the graph G. • The slope of the path between 2 nodes, based on distance and altitude differences of the peaks. • The distance and direction of the next intersections can optionally be used to determine spatial and / or temporal proximity for future longitudinal realignment. - semantic properties with: • The class nature of the edge between vertices (roads, paths) • Confidence, expressed as a real number on [0, 1], of belonging to the class indicated. - topological properties with: • The nature of a junction between edges of the graph G by connectedness analysis describes its geometric shape class (like L, O, T, X, Y cf fig 3). • The topological signature corresponding to the different angular differences between branches of the graph G and in particular at the level of a crossroads.

[0064] Memory 22 also includes, associated with graph G, the coordinates of the characteristic elements represented on graph G (latitude, longitude, preferably altitude). The accuracy errors associated with these data are preferably also provided; they are obtained in particular from the quality of the source data used to construct the graph.

[0065] In an example of implementation, the graph G in memory 22 was previously obtained by processing one or more ortho-images of the navigation area so as to highlight the characteristic elements of the navigation area on the ortho-image(s) and to extract the shape of said characteristic elements to form the graph G.

[0066] Preferably, the characteristic features are highlighted on the orthoimage(s) by a deep learning algorithm, for example, an algorithm implemented by a neural network. In one example implementation, the deep learning algorithm provides grayscale images indicating the confidence of finding routes with certainty or, conversely, the possibility of finding routes with the lowest confidence. The grayscale image thus obtained can be represented in logarithmic form in order to best highlight the less clear access routes on the orthoimage. The resulting image of the routes is then suitable for various classic image processing techniques, for example morphological, connectivity, skeletonization or learning, in order to construct a graph G.

[0067] In one example of implementation, the graph G is developed before the first movement of the optronic system 18, for example using an image seen from the sky already available on the scale of the planet, or a part of an image that would have been acquired more recently by an aerial vector having an optronic sensor and a means allowing the realization of geo-referenced ortho-images on the area of ​​evolution of the optronic system 18. This ability to construct the graph G, including in short loop, gives a dynamic aspect to the application.

[0068] In one implementation example, the graph G was consolidated using, for example, several images, including multi-date / multi-season images. The merging of several graphs still constitutes a graph that is used in the same way as a graph obtained from an instance of a single modality.

[0069] Preferably, memory 22 includes ephemeris files giving the position of the sun or an astrometric data catalogue giving the position of other bodies in the solar system according to the dating.

[0070] Preferably, the memory 22 also includes a geometric image capture model. This image capture model makes it possible to associate a pixel of one of the images, produced by the optronic system 18, with a spatial direction described by two angles (bearing, elevation) in the axis system of the frame attached to the optronic system 18. This frame of the optronic system 18 has an attitude relative to the local geographic frame, and the three (Euler) angles characterizing this attitude constitute three of the six unknowns of the complete pose of the system to be determined. Thus, any point of a reference element of the scene having coordinates in an optronic image has a relative direction in the frame of the optronic system 18.

[0071] The computing unit 28 is suitable for receiving data from other elements of the optronic system 18, in particular images from the imaging device 21, data stored in memory 22 and data from the navigation device 20.

[0072] The computing unit 28 is, for example, a processor.

[0073] In one example, the computing unit 28 interacts with a computer program product that includes an information carrier. The information carrier is a carrier readable by the computing unit 28.

[0074] The readable information medium is a medium suitable for storing electronic instructions and capable of being connected to a bus of a computer system. For example, the readable information medium is an optical disc, a CD-ROM, a magneto-optical disc, a ROM, a RAM, an EPROM, an EEPROM, a magnetic card, an optical card, or a key. USB. The computer program product, including program instructions, is stored on the information medium.

[0075] The computer program can be loaded onto the computing unit 28 and leads to the implementation of steps of an automatic recalibration process of the navigation of an optronic system 18 evolving in a navigation area, when the computer program is implemented on the computing unit 28 as will be described in the rest of the description.

[0076] The display element 30 is suitable for displaying images from the imaging device 21 and / or data stored in memory 22.

[0077] The display element 30 is, for example, a display, such as an OLED screen.

[0078] The operation of the optronic system 18 leading to the implementation of an automatic recalibration process for the navigation of an optronic system 18 evolving in a navigation zone, will now be described with reference to the flowchart of [Fig.4], and to [Fig.5] which illustrates an example of a step of the process.

[0079] The recalibration process is implemented by the optronic system 18.

[0080] The steps of the process are implemented over time (real time). From Preferably, the different stages of the process are implemented continuously, for example at the acquisition frequency of the imaging device 20.

[0081] At initialization, the position and orientation of the optronic system 18 are, for example, provided to the navigation device 20. For example, the initial position and initial orientation are given either by a GNSS (if it is reliable at the location of the start of the mission), or by bearings from georeferenced landmarks, or by any other means of static positioning of the bearing type, for example.

[0082] ESTIMATE OF A STANDARD INSTALLATION

[0083] The method includes a step 90 of estimating, by the navigation device 20, a current position and a current orientation for the optronic system 18. These estimates are subject to errors depending on the drift of the navigation device 20. It is this drift that we seek to correct in the following steps of the method.

[0084] IMAGING DATA ACQUISITION

[0085] The method includes a step 100 of acquiring imaging data by the imaging device 21.

[0086] The imaging data is an image or a stream of images of the environment seen from the optronic system 18.

[0087] Preferably, the image data is a panoramic (or quasi-panoramic) image, or a stream of panoramic (or quasi-panoramic) images of scene 10.

[0088] The acquisition can take place with the carrier stationary or in motion. The speed of motion must be compatible with the integration time of the imaging device to avoid introducing blur into the image. For example, a speed of approximately 50 km / h can be achieved with an integration time of a few milliseconds. Preferably, the position maintained by the inertial measurement unit will be acquired simultaneously with the image acquisition.

[0089] In one example, the image data is constructed from videos of optronic sensors distributed around the vehicle 19. The images from said sensors are time-stamped and assembled using beam-fitting processing to establish a panoramic image or a stream of panoramic images of the scene 10.

[0090] Knowledge of the image arrangement, describing their position and attitude in a common reference frame to the vehicle, can be refined as needed to meet a performance objective. For this purpose, preference is given to using observations consisting of: - Matching of image primitives in the overlap areas between images; to improve accuracy, small point primitives with sub-pixel accuracy are used. - Matching objects with known ground coordinates with their corresponding image coordinates; - A mixture of the two previous types of observations: relative and absolute.

[0091] IMAGING DATA PROCESSING

[0092] The method includes a step 110 of processing the imaging data by the computing unit 28.

[0093] Processing step 110 includes segmenting the imaging data so as to highlight and extract characteristic elements, called identified elements, from the imaging data.

[0094] In one example implementation, segmentation is performed by a deep learning algorithm. Preferably, segmentation is performed by a semantic segmentation algorithm.

[0095] The segmentation is, for example, a class-based segmentation, including a lane class, where lanes are, for example, roads, highways, paths, or trails. The outlines of other vehicles detected during this segmentation can be used as potential road masking information, and the semantic information can be exploited on several successive images.

[0096] Other classes of elements can also be used, for example, building classes, ground element classes, tree / forest classes, pole / pylon classes, these classes being "static" and therefore likely to be present on an orthophoto taken from the sky. Still other classes can be considered, by example a sky class not present on an ortho image taken from the sky, and vehicle and human "mobile" classes probably not present on a previously taken ortho image of the sky.

[0097] The extraction is, for example, carried out by an algorithm for extracting the contours of the characteristic elements. For example, the contours of the characteristic elements correspond to the intersections between each highlighted characteristic element and the highlighted elements for different classes of elements (road, field, vegetation, buildings...).

[0098] An example of segmentation is illustrated in the top image of [Fig.5] and of extraction in the bottom image of [Fig.5].

[0099] In particular, each identified characteristic element has a direction which is obtained as a function of the pixel coordinates of the characteristic element on the panoramic data since each pixel corresponds to a direction characterized by its angular values ​​(bearing, site).

[0100] DETERMINATION OF A SET OF RECALIBRATION CORRECTIONS

[0101] The method includes a step 120 of determining, by the computing unit 28, a set of recalibration corrections for the navigation device 20.

[0102] The determination step includes matching at least one identified element on the image data with a characteristic element of the graph G as a function of the current position and current orientation of the optronic system 18.

[0103] In an example implementation, the matching process involves searching for characteristic elements of the graph G, called candidate elements, within a predetermined perimeter around the current position. The element associated with the identified element is chosen from among the candidate elements according to a selection criterion.

[0104] In an example implementation, the selection criterion stipulates that the associated element is the candidate element closest to the current position.

[0105] In an example implementation, the distance separating the current solution from nearby branches of the graph G may present an ambiguity. Thus, as a complement or alternative, when several candidate elements are present at a distance less than a predetermined distance from the current position, the selection criterion stipulates that the associated element is the candidate element generating a lower association cost when implementing a probabilistic approach.

[0106] Preferably, the probabilistic approach implements a calculation of Mahalanobis distances between each candidate element and the current position.

[0107] More specifically, the following substeps are implemented, for example: - A windowing substep, based on the quality of the current solution, allows the search for vertices of the graph G that are compatible in distance, this which allows for the identification of summits and edges suitable for vehicle traffic. - We deduce the distances of the candidate edges to the current position solution, and also consider the orientations of these edges, as well as the errors associated with these quantities. - We use a probabilistic data association (PDA). The decision to associate the current position with one of the candidate branches of the graph G is made using a Mahalanobis distance that takes into account both the distances of the branches of the graph G to the solution. Each candidate edge 'G' of the graph for finding the vehicle is assigned a cost CG, which is the sum of the normalized distances and orientations. [01081 co = (Pc, - PN) (Po - P^) + (Va - VN) rAÿ (Va - VN)

[0109] Expression in which: - Pg represents one of the candidate graph positions, obtained by projection perpendicular to the edge of the current navigation position Ph - AP corresponds to the position covariance, the sum of the navigation position error covariance indicated by the Apv filter and the error covariance of the vehicle's position that would move along this edge Ag. The latter inherits in particular the metadata of the geographic products used: planar position error of the ortho-image and vertical error for the DEM. - VG represents the error on the angular orientation of the candidate branch G of the graph, and VN the orientation of the velocity vector current navigation solution. - AzV corresponds to the orientation covariance, which is the sum of the navigation orientation error covariance Ay, directly calculated from the variance on the current velocity vector of the filter, and the covariance Api describing the orientation error of the candidate edge of the graph. The latter is obtained, in particular, from the metadata of the geographic products used.

[0110] Among all the candidate associations, the simplest approach then selects the one with the lowest cost. To mitigate possible association ambiguities, a more complex approach maintains a number of possible association hypotheses over time by making the final assignments to the graph data as soon as a doubt is resolved at the threshold crossing in distance / orientation. If doubt persists, the system can raise an alarm.

[0111] Preferably, the matching is also a function of the digital terrain model of the navigation area, stored in memory 22.

[0112] The determination step 120 also includes the determination of a set of corrections to the current position and current orientation of the optronic system 18, to realign at least one identified element on the imaging data with the corresponding characteristic element of the graph G.

[0113] In one embodiment, at least one transverse realignment correction is determined for the current position and orientation of the optronic system 18 relative to the identified element under consideration. Assuming that the optronic system 18 is moving along a track, this allows for realignment of the current position and orientation on the track, particularly in the event of deviation off the track.

[0114] In one implementation example, when at least one identified element is a track intersection and / or a track with a specific curvature, the matching is also performed based on the shape of the identified element. The shape incorporates, for example, the curvatures of the characteristic element. The correction set includes at least one longitudinal realignment correction for the current position and orientation of the optronic system 18 relative to the identified element in question.

[0115] As an example of application, on the majority of these journeys a wheeled military vehicle will be traveling on asphalt, and the processing will determine its transverse position relative to the road based on the position of the road edges in the sensor image. In practice: - Relatively, the road located in front of and behind the vehicle is always positioned around the same places on 2 branches at 180° in the panoramic image, this presence confirms the vehicle's movement on a ground having the semantic segmentation class. - In absolute terms, we have an approximate orientation based on the branch of the graph G traversed a priori, as well as an absolute orientation based on the velocity vector of the current navigation solution. These two corresponding directions should be close, within errors.

[0116] When approaching a graph edge G with a very pronounced radius of curvature, it becomes relevant to monitor and estimate the curvature of the road edges in front of the vehicle. This mechanism provides a longitudinal realignment correction to the road, except in the presence of a lane intersection.

[0117] NAVIGATION UPDATE AND MAINTENANCE

[0118] The method includes a step 130 of updating and maintaining the navigation of the navigation device 20 according to the set of corrections determined.

[0119] Navigation is maintained automatically, i.e. without requiring intervention from the crew or occupants of vehicle 19.

[0120] The update consists, for example, of correcting the position and orientation of the optronic device according to the set of corrections determined (angular deviation correction in particular).

[0121] The quality of navigation is indicated by the state covariance of the filter.

[0122] As an optional addition, the navigation filter is also powered by correspondences between directions provided by the imagery data using: - relative observations between images which can be associated by classical techniques to construct correspondences between images usable according to classical approaches of the type visual odometry, Motion From Structure (MFS) or Simultaneous Localization and Mapping (SLAM) to feed the filter with correspondences of the type of spatial directions on common fixed objects of the scene. - absolute observations with a panoramic view from a site high enough to acquire images of celestial bodies such as the moon or the sun and carrying their ephemerides.

[0123] For example, when the method detects a separation of the edges of graph G where its information becomes unusable, the re-registering is then limited to the use of celestial bodies or odometric measurements of the video to maintain the navigation filter fed by IMU measurements as closely as possible. In this situation, the re-engagement of graph G is also detected in order to resume the transverse and longitudinal re-registering of the lanes.

[0124] Thus, the default operation uses horizontal and vertical position corrections. The vehicle's altitude can also be estimated based on its height above the road and the road's altitude obtained from the horizontal coordinates of a DEM if available, or more generally by interpolating the vertical information carried by the graph's nodes.

[0125] Updating and improving a position in an inertial filter of the vehicle's CNI also improves the vehicle's attitude calculation. Improving this attitude information at high rates facilitates the association decision by comparing the orientation contribution to those of the graph segments.

[0126] Optionally, an additional phase is added for consistency checking. This analyzes the consistency of the IMU data: - Analysis / monitoring of inertial data and jump detection: • detection of abrupt jump-type inconsistencies in kinematic data, • detection of stop and start by analyzing accelerations / speeds. - Exploitation of inertial measurements and consistency with graph G and image data: • Plane accelerations for turn detection and acceleration-radius of curvature consistency. • longitudinal attitudes for consistency with the terrain or consistency of the vehicle's inclination according to its attitude with respect to the terrain.

[0127] Preferably, in the absence of identified features in the image data, the steps of determining a set of repositioning corrections and updating and maintaining the navigation are interrupted, these steps being resumed as soon as at least one feature is identified in the image data. In an example implementation, if a panoramic image does not reveal lanes in front of / behind the vehicle, the vehicle is declared as moving outside the graph. This state allows the processing to be adapted and a procedure to be launched that will evaluate and confirm its approach to the graph during its future movements.

[0128] Thus, when off-road the optronic system makes it possible to detect that the vehicle leaves the road and therefore the graph, and in this case to rely on the navigation device 20 (the inertial unit) for navigation and to predict the next intersection with the graph (crossing of road or path) allowing a recalibration on this occasion.

[0129] At the end of the navigation update and maintenance step, at least the corrected position of the optronic system 18 is stored in the memory 22 of the optronic system 18 for later reuse in the event of an interruption of the steps for determining a set of recalibration corrections and for updating and maintaining the navigation. This can also be used for debriefing, evaluation, or fine-tuning phases during or after the mission.

[0130] ADVANTAGES AND CONCLUSION

[0131] The invention solves the problem of recalibrating the navigation of a military vehicle without GNSS, by using a semantic segmentation of an optronic vision of the scene and by appropriately recalibrating the information extracted from the images to a graph G of the on-board channel network and representing as best as possible the information that can be automatically extracted from the environment.

[0132] The crew is thus exempt from manually intervening every few tens of kilometers if it has a control unit with a drift of 1m / km and a setpoint of 10m maximum deviation for example.

[0133] The proposed solution allows operation in structured environments, but also in unstructured environments; that is to say, including outside urban areas and outside of paved roads such as paths, tracks in desert environments, fields....

[0134] The vehicle is limited to passive sensor equipment for reasons of electromagnetic discretion and energy consumption. The usable sensors are limited to inertial components and an optronic sensor providing video of the vehicle's environment.

[0135] In particular, the process analyzes the graph G and dresses it with geometric (e.g. road width), topological (e.g. intersection with road junction having an L, O, T or X shape) and semantic (e.g. road, path, track) properties.

[0136] In particular, in navigation the optronic sensor acquires a high frame rate video of the landscape and the images are subjected to semantic classification by means of another neural network allowing to infer classes of the environment and in particular classes such as roads, paths, grass, sidewalk.

[0137] Preferably, a high frequency (HF) analysis makes it possible to locate and characterize the rolling area of ​​the vehicle with respect to the elements of the graph G and to locate the vehicle transversely at high speed.

[0138] Preferably, a low-frequency (LF) analysis is requested at the junction approach or relevant features to locate the vehicle longitudinally to the road / path. This sequencing is performed according to the local analysis of the graph topology G using the vehicle's position.

[0139] Image observations and graph G thus make it possible to determine visual positions and orientations of the vehicle to correct and maintain the navigation filter.

[0140] A person skilled in the art will understand that the embodiments described above can be combined to form new embodiments provided that they are technically compatible.

Claims

1. Demands Method for automatically recalibrating the navigation of an optronic system (18) evolving in a navigation zone, the navigation zone comprising characteristic elements, the characteristic elements comprising channels, the optronic system (18) comprising the following elements integrated into the optronic system (18): - a navigation device (20) capable of estimating over time at least one position and orientation for the optronic system (18) and of guiding the movement of the optronic system (18) according to the estimated position and orientation, - an imaging device (21) capable of acquiring images of the environment seen from the optronic system (18), - a memory (22) in which a graph is stored (G) characteristic features of the navigation area as seen from the air, - a calculation unit (28), the process being implemented by the optronic system (18) and comprising over time the following steps: - the estimation, by the navigation device (20), of a current position and a current orientation for the optronic system (18), - the acquisition of imaging data by the imaging device (21), the imaging data being an image or a stream of images of the environment seen from the optronic system (18), - the processing of the imaging data by the computing unit (28), the processing step including the segmentation of the imaging data so as to highlight characteristic elements, called identified elements, on the imaging data, - the determination, by the calculation unit (28), of a set of corrections for recalibrating the navigation device (20), the determination step comprising: • the matching of at least one identified element on the image data with a characteristic element of the graph (G) as a function of the current position and current orientation of the optronic system (18), • the determination of a set of corrections of the current position and current orientation of the optronic system (18), to realign the at least one identified element on the image data with the corresponding characteristic element of the graph (G), and - the updating and maintenance of the navigation of the navigation device (20) as a function of the set of corrections determined.

2. A method according to claim 1, wherein during the step of determining a set of corrections, at least one transverse recalibration correction of the current position and orientation of the optronic system (18) relative to the identified element under consideration is determined.

3. A method according to claim 1 or 2, wherein the characteristic elements of the graph (G) include channel crossings and / or channels with specific curvatures, during the step of determining a set of corrections, when at least one identified element is a channel crossing and / or a channel with a specific curvature, the matching is also carried out according to the shape of the identified element, and the set of corrections includes at least one longitudinal recalibration correction of the current position and orientation of the optronic system (18) with respect to the identified element considered.

4. A method according to any one of claims 1 to 3, wherein during the step of determining a set of corrections, the matching includes searching for the characteristic elements of the graph (G), called candidate elements, within a predetermined perimeter around the current position, the element associated with the identified element being chosen from among the candidate elements according to a selection criterion.

5. A method according to claim 4, wherein the selection criterion stipulates that: - the associated element is the candidate element closest to the current position, or - in the presence of several candidate elements at a distance less than a predetermined distance from the current position, the associated element is the candidate element generating a lower association cost when implementing a probabilistic approach, preferably the probabilistic approach implementing a calculation of Mahalanobis distances between each candidate element and the current position.

6. A method according to any one of claims 1 to 5, wherein the graph (G) in memory (22) has been previously obtained by processing one or more ortho-images of the navigation area so as to highlight the characteristic elements of the navigation area on the ortho-image(s) and to extract the shape of said characteristic elements to form the graph (G).

7. A method according to claim 6, wherein the characteristic features are highlighted on the ortho-image(s) by a deep learning algorithm, for example an algorithm implemented by a neural network.

8. A method according to any one of claims 1 to 7, wherein a digital terrain model of the navigation area is also stored in the memory (22) of the optronic system (18), during the step of determining a set of corrections, the matching also being a function of the digital terrain model.

9. A method according to any one of claims 1 to 8, wherein in the absence of identified elements on the image data, the steps of determining a set of registration corrections, and of updating and maintaining navigation are interrupted, said steps being resumed as soon as at least one element is identified on the image data.

10. A method according to any one of claims 1 to 9, wherein, at the end of the navigation update and maintenance step, at least the corrected position of the optronic system (18) is stored in the memory (22) of the optronic system (18) for reuse later in the event of a possible interruption of the steps of determining a set of recalibration corrections, and of updating and maintaining the navigation.

11. Optronic system (18) evolving in a navigation zone, the navigation zone comprising characteristic elements, the characteristic elements comprising channels, the optronic system (18) being configured to implement a method according to any one of claims 1 to 10, the optronic system (18) comprising the following elements: - a navigation device (20) adapted to estimate over time at least one position and orientation for the optronic system (18) and to guide the movement of the optronic system (18) according to the estimated position and orientation, - an imaging device (21) adapted to acquire images of the environment seen from the optronic system (18), - a memory (22) in which is stored a graph (G) of the characteristic elements of the navigation zone in view of the sky, and - a computing unit (28).

12. Vehicle (19), such as a land vehicle, comprising an optronic system (18) according to claim 11.

Images