Method for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment

Abstract

Method for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment The invention relates to a method (40) for segmenting image(s) and locating characteristic elements of a landing environment within radar image(s) comprising: - a supervised learning phase (48) of a deep learning tool comprising: - obtaining (50) a set of labeled learning data; - training (52) said tool on said set, said tool combining at least one attention mechanism and a convolutional neural network, - an inference phase (58), applying said trained tool to a current radar image, comprising: - segmenting (60) said current radar image by determining whether or not a pixel belongs to one of the characteristic elements corresponding to a class of said segmentation;- the location (62) of at least one of the characteristic elements in the reference frame of said radar using the result of said segmentation. Figure for the abstract: Figure 2;

Description

Title of the invention: Method for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment

[0001] The present invention relates to a method for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from M-dimensional radar measurements, with 2 < M < 4, said aircraft carrying at least one radar.

[0002] The invention also relates to a computer program comprising software instructions which, when implemented by a programmable electronic device, implement such a method.

[0003] The invention also relates to an electronic device for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from M-dimensional radar measurements, with 2 < M < 4.

[0004] The invention also relates to a landing assistance system comprising at least one radar and such a device.

[0005] The invention also relates to an aircraft comprising such a landing aid system.

[0006] The invention lies in the field of aircraft navigation. In this field in particular, it is known to equip aircraft with landing aid devices capable in particular of determining the position of the aircraft relative to the landing runway.

[0007] In particular, to land in conditions of reduced visibility, in particular in the presence of fog, snow, heavy rain, etc., the crew or pilot of an aircraft generally relies on on-board equipment corresponding to such landing aid devices including, in a known manner, instrument landing systems (ILS), microwave landing systems (MLS), a GPS receiver making it possible in particular to obtain a geolocated position of the aircraft, etc.and also relies on ground-based landing aid infrastructure such as approach light ramps, a PAPI (Precision Approach Path Indicator) approach slope indicator, markers used by the ILS instrument landing system such as the runway centerline, the oblique glide path leading to the runway, the runway threshold, the runway itself, the touchdown zone, etc.

[0008] However, when the visibility and / or radio navigation conditions are insufficient, such conventional solutions are limited because they do not offer sufficient precision to comply with the air navigation safety objectives applied to the different types of approach and landing, or require one or more specific infrastructures that are costly or impossible to implement regardless of the landing zone.

[0009] The aim of the invention is then to propose a solution providing, to conventional location solutions, a complement of integrity and / or continuity and / or precision, such a solution making it possible to ensure automatic aircraft guidance which is compliant with the air navigation safety objectives applied to the different types of approach and landing.

[0010] To this end, the invention relates to a method for segmenting landing area(s) and locating predetermined characteristic elements of an aircraft landing environment within landing area(s) obtained from M-dimensional radar measurements, with 2 < M < 4, said aircraft carrying at least one radar, said method comprising:

[0011] - a supervised learning phase of a deep learning tool including:

[0012] - obtaining a labeled training data set comprising images, obtained from M-dimensional radar measurements previously acquired during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar;

[0013] - training said deep learning tool on said data set labeled learning tools, said deep learning tool combining at least one attention mechanism and a convolutional neural network, said training being based on an iterative calculation aimed at minimizing a cost function quantifying the difference between the prediction of said deep learning tool and the associated ground truth,

[0014] - an inference phase, applying said trained deep learning tool to a current radar image obtained, from at least one current radar measurement, in the approach phase before landing of the aircraft, comprising:

[0015] - the segmentation of said current radar image by said learning tool deep trained by determining whether or not a pixel of said current radar image belongs to one of the predetermined characteristic elements of the landing environment, each of said characteristic elements corresponding to a class of said segmentation;

[0016] - the location of at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar using the result of said segmentation.

[0017] Thus the present invention aims to take advantage of images obtained from radar measurements, in M ​​dimensions, for example millimeter radar measurements.

[0018] Indeed, it is certainly known to equip an aircraft with a radar system comprising at least one directional antenna, adapted to emit at least one beam of radiofrequency waves along a controllable direction sighting axis, said direction being defined by an elevation angle and a bearing angle, and for example oriented towards the front of the aircraft.

[0019] However, advantageously the present invention proposes a specific exploitation of M-dimensional radar measurements, in particular millimeter radar measurements, by applying to them a deep learning tool based in particular on attention mechanisms to improve the detection, by segmentation, of the characteristic elements of the approach scene during the landing phase of the aircraft.

[0020] Thus for such a particular application of detection, by segmentation, of the characteristic elements of the approach scene during the landing phase of the aircraft, the present invention proposes to benefit from the power of learning by artificial intelligence by means of said deep learning tool based in particular on the attention mechanisms applied in a non-trivial manner specifically to radar measurements, in particular millimeter radar measurements, in M ​​dimensions.

[0021] According to the present invention, the deep learning tool is previously trained on a labeled training data set comprising images, obtained from radar measurements, in particular millimeter radar measurements, with M dimensions previously acquired during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar.

[0022] In other words, the prior training provides a self-attentive artificial intelligence model capable of highlighting, during the inference phase, the various characteristic elements visible and relevant on the current radar measurement by enhancing the contrast and / or homogenizing an area.

[0023] According to other advantageous aspects of the invention, the method for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from radar measurements, in particular millimeter radar measurements, in M ​​dimensions, comprises one or more of the following characteristics, taken in isolation or in all technically possible combinations:

[0024] - said deep learning tool further comprises a predetermined number of LSTM recursive cells,

[0025] said LSTM recursive cells receiving as input said radar measurements grouped in time series, the acquisition frequency of each radar measurement of a time series being predetermined, the output of said LSTM recursive cells being provided as input to said at least one attention mechanism whose output is provided as input to said convolutional neural network

[0026] or said at least one attention mechanism receiving as input said radar measurements grouped in time series, the acquisition frequency of each radar measurement of a time series being predetermined, the associated outputs of said at least one attention mechanism being provided as input to said recursive cells whose respective outputs are provided as input to said convolutional neural network;

[0027] - the acquisition step between each radar measurement of said time series is greater than or equal to half a second, and wherein the total duration of said time series is less than or equal to two seconds;

[0028] - said at least one attention mechanism comprises a number of masks greater than or equal to the number of classes of predetermined characteristic elements of the landing environment;

[0029] - said at least one attention mechanism comprises the coding of at least one of the deciphering the relative position, in the spatial or temporal domain, of said predetermined characteristic elements with respect to each other within said landing environment;

[0030] - said at least one attention mechanism is:

[0031] - a self-attention mechanism capable of translating the similarity of different patterns into within the same radar measurement or time series; or

[0032] - a specific attention mechanism capable of associating areas of the same radar image to a set of predefined patterns representative of previously available information on the landing environment;

[0033] - said deep learning tool is configured to apply a multi- classes by first using a multi-channel labeling of said training data set according to which a pixel is capable of belonging to several classes of said predetermined characteristic elements;

[0034] - the method further comprises a step of providing said segmentation result and / or of said location to a pilot of said aircraft and / or to automatic pilot equipment of said aircraft,

[0035] and / or said location of at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar corresponds at least to the location of the landing runway and comprises:

[0036] - a first post-processing comprising:

[0037] - the determination of at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance and circular, the circular of a point corresponding to the angle formed from the origin of said reference frame and the line of sight of the radar to the projection of said point in the plane (0, x, y) and oriented positive following the rotation around the axis (0, z); and

[0038] - determining the orientation of the axis of the landing runway in the reference frame of said radar, by:

[0039] - correlation between the result of said segmentation and a known contour of said track; and / or

[0040] - identification of track edges by determining a plurality of points of interest whose contour contrasts in said current radar image by being representative of a track rectangle whose lines are obtained by linear regression from said plurality of points,

[0041] or

[0042] - a second post-processing comprising the determination, by neural network, of said at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance and circular, and of said orientation of the axis of the landing runway in the reference frame of said radar;

[0043] - each radar measurement is preprocessed before use during the learning phase and / or inference, the preprocessing of each radar measurement including:

[0044] - filtering retaining only radar measurements having a distance less than 2200m, and / or

[0045] - a dimension reduction of each radar measurement to a two-dimensional radar measurement sional or three-dimensional

[0046] The invention also relates to a computer program comprising software instructions which, when executed by a computer, implement a method for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from radar measurements, in particular millimeter radar measurements, with M dimensions as defined above.

[0047] The invention also relates to an electronic device for segmenting landing area(s) and locating predetermined characteristic elements of an aircraft landing environment within landing area(s) obtained from radar measurements, in particular millimeter radar measurements, with M dimensions, with 2 < M < 4, said aircraft carrying at least one radar, in particular millimeter radar, said device being configured to implement:

[0048] - a supervised learning phase of a deep learning tool including:

[0049] - obtaining a labeled training data set comprising images, obtained from M-dimensional radar measurements previously acquired during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar;

[0050] - training said deep learning tool on said data set labeled learning tools, said deep learning tool combining at least one attention mechanism and a convolutional neural network, said training being based on an iterative calculation aimed at minimizing a cost function quantifying the difference between the prediction of said deep learning tool and the associated ground truth,

[0051] - an inference phase, applying said trained deep learning tool to a current radar image obtained, from at least one current radar measurement, in the approach phase before landing of the aircraft, comprising:

[0052] - the segmentation of said current radar image by said learning tool deep trained by determining whether or not a pixel of said current radar image belongs to one of the predetermined characteristic elements of the landing environment, each of said characteristic elements corresponding to a class of said segmentation;

[0053] - the location of the predetermined characteristic elements of the environment landing in the reference frame of said radar using the result of said segmentation.

[0054] The invention also relates to a landing assistance system comprising at least one radar, in particular millimetric radar, and such an electronic device for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from radar measurements, in particular millimetric radar measurements, in M ​​dimensions, with 2 < M < 4.

[0055] The invention also relates to an aircraft comprising such a landing assistance system.

[0056] The invention will appear more clearly on reading the description which follows, given solely by way of non-limiting example, and made with reference to the drawings in which:

[0057] [Fig-1] [Fig.l] is a block diagram of the main functional blocks of a electronic device for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from radar measurements, in particular millimeter radar measurements, in M ​​dimensions;

[0058] [Fig.2] [Fig.2] is a flowchart of the main steps of a seg process processing of image(s) and localization of predetermined characteristic elements of an aircraft landing environment within image(s) obtained from radar measurements, in particular millimeter radar measurements, in M ​​dimensions according to one embodiment of the invention.

[0059] [Fig.l] illustrates an embodiment of an electronic device 10 for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment according to an embodiment of the invention.

[0060] Such a device is suitable for being integrated within a landing aid system, not shown, comprising at least one radar, in particular millimetric radar, not shown, said landing system itself being suitable for being carried on board an aircraft, not shown.

[0061] More precisely, the device 10 according to the present invention is capable of implementing an artificial intelligence AI according to a supervised learning phase of a deep learning tool applied to a set of labeled learning input data E comprising images, obtained from radar measurements, in particular millimeter radar measurements, with M dimensions acquired beforehand during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar.

[0062] Following said learning phase, the device 10 according to the present invention is capable of implementing an inference phase applying said trained deep learning tool to a current radar image obtained, from at least one input E corresponding this time to at least one radar measurement, in particular at least one millimeter measurement, current, in the approach phase before landing of the aircraft.

[0063] To do this, the electronic device 10 according to the present invention therefore firstly comprises a module for obtaining, not shown in [Fig.l], said set of labeled learning input data E used during the learning phase, and also at least one input E corresponding this time to at least one radar measurement, in particular at least one millimeter measurement, currently used during the inference phase.

[0064] The input data set E comprises as inputs radar measurements, in particular millimeter radar measurements, of 2 to 4 dimensions. Each dimension corresponds respectively, in the reference frame of said radar, to:

[0065] - a distance measurement from the radar to a point, or

[0066] - a circular measurement, the circular of a point corresponding to the angle formed from from the origin of said reference point and the radar sighting axis to the projection of said point in the plane (0, x, y) and oriented positively following the rotation around the axis (0, z), or

[0067] - a Doppler measurement, or

[0068] - an elevation measurement, the elevation of a point corresponding to the angle formed at from the origin of said reference point and the radar line of sight to the projection of said point in the plane (0, x, z) and oriented positively following the rotation around the axis (0, y).

[0069] Such measurements are acquired by a radar, in particular millimeter radar, and processed in real time during the inference phase by the previously trained deep learning tool. The “characteristics” of the different dimensions, in particular their sizes, their amplitudes and their resolutions, depend on the characteristics of the radar used, in particular a millimeter radar.

[0070] According to the embodiment of [Fig. 1], it should be noted that the input data are provided in the form of time series, a time series comprising a series of data, for example five data as represented in [Fig. 1] associated respectively with five successive instants of distinct radar captures tN 4, tN.3, tN 2, tN ,ct t N along the time axis t.

[0071] According to the example of [Fig.l], the device 10 therefore comprises a deep learning tool 12 comprising at least: an attention mechanism 14 and a convolutional neural network.

[0072] In particular, according to the example of [Fig.l], to exploit the temporal information and promote detection consistency, by segmentation, the deep learning tool 12 further optionally comprises a predetermined number of recursive LSTM cells 16.

[0073] According to the embodiment of [Fig.l], said at least one attention mechanism 14 is configured to receive as input said radar measurements, in particular millimeter radar measurements, grouped in a time series, the acquisition frequency of each radar measurement, in particular a millimeter radar measurement, of a time series being predetermined, the associated outputs of said at least one attention mechanism 14 being provided as input to said recursive cells 16, the respective outputs of which are provided as input to said convolutional neural network 18.

[0074] In other words, according to the embodiment of [Fig.l], the deep learning tool 12 comprises a module for implementing at least one attention mechanism 14, connected to an optional module for implementing a predetermined number of recursive LSTM cells 16, connected to a module for implementing a convolutional neural network 18. Thus, according to this embodiment, each instant of the time series has an attention mechanism applied individually, before being processed as a series by the convolutional LSTM cells.

[0075] Subsequently, each of the modules of the deep learning tool 12 according to the present invention is detailed.

[0076] First of all, the module for implementing at least one attention mechanism 14 is described below. In particular, it should be noted that, in general (i.e. without any application to the particular technical field of the present invention, namely aircraft navigation), attention mechanisms are notably described by Vaswani, Ashish, et al. in the article entitled "Attention is ail you need." Advances in neural information processing Systems 30 (2017) and are based on the following product (of English ScaledP roduct Attention): ^en^onÇQ y — softmaxi)V' \ / also called attention mask, with Q: the query which represents the information that we wish to associate, and which will also be compared, with each of the keys K (from the English key), in order to associate a similarity score with each of these associations in the form of an attention filter: „ / qkt \, with dk the softmaxy —,— ^dk dimension of Q and K. This filter is applied to the values ​​V (from the English value) which contain the information associated with the keys K so that a query is defined only by its similarities to the values.

[0077] According to the present invention, for specific application to the field of aircraft navigation of these attention mechanisms, Q corresponds to the query, namely the information contained in each radar measurement. The information is fragmented and encoded by descriptors detailed below; V and K correspond to the values ​​and keys to which the queries are compared and associated, their definition varying according to two application contexts considered as also detailed below.

[0078] According to the present invention, the use of attention mechanisms for the localization of characteristic elements in radar measurement has the advantage of analyzing patterns and associating the different patterns present in the images obtained from radar measurements, in particular millimeter radar measurements, with M dimensions.

[0079] In the context of observation of an aerodrome where the arrangement of the characteristic elements and the architecture of the characteristic elements themselves are codified, such pattern motifs, with a certain variability, remain in fact constant from one landing scene to another (example of an aerodrome) and therefore benefit from being exploited and associated for the interpretation of the geometry of the scene, so that taking into account attention mechanism(s) within the deep learning tool 12 according to the present invention is relevant and advantageous.

[0080] As an optional addition, said at least one attention mechanism comprises a number of attention masks greater than or equal to the number of classes of predetermined characteristic elements of the landing environment.

[0081] More precisely, in the case of an application to a radar measurement, in particular a millimeter radar measurement, these associations are represented by masks attention on the areas of the radar reference frame relevant to the desired image segmentation and the associated detection of characteristic elements, such characteristic elements of the segmentation result R_S, correspond in particular according to the application of the present invention to the ramp R, the track P, the set L of lamps of the ramp, the axis A of the track, the threshold S of the track, etc., an attention mask making it possible to define, in a radar image obtained from the radar measurements, a large area called "attention area" composed of several small areas associated by their similarities. For example, the pixels of the track will be linked so as to form a first area, the pixels of the ramp will be linked so as to form another second area, the pixels of track contours will be linked so as to form another third area, etc.

[0082] The number of masks defined by the model is a variable parameter to be defined by the user according to the context, when it is multiple we then speak of Multi-Head Attention. Advantageously the number of masks is greater than the number of distinct classes of predetermined characteristic elements present in the approach scene of the landing environment, because if the objective is to “discretize” N characteristic elements to locate them, the attention mechanism is more effective when it defines at least N areas of interest.

[0083] According to a particular optional aspect, as generally introduced in the aforementioned article, the attention mechanisms implemented according to the present invention include a positional anchoring (from the English positionnai embedding) which makes it possible to encode the information of the relative position of the characteristic elements with respect to each other, whether this concerns the spatial or temporal domain according to the protocol used.

[0084] In other words, as an optional addition, said at least one attention mechanism comprises the coding of at least one descriptor of the relative position, in the spatial or temporal domain, of said predetermined characteristic elements with respect to each other within said landing environment.

[0085] The arrangement and architecture of the characteristic elements is thus codified via said at least one descriptor which is therefore then determining information.

[0086] More precisely, unlike optical images, it should be noted that the use of radar, in particular millimeter radar, allows the evaluation of distances in the radar plane. The characteristic elements of the approach scene to be located having for the most part a codified architecture and arrangement, such as the spacing between two lamps of the ramp or the edge of the runway for example, according to the present invention descriptors adapted to these expected patterns are defined so as to discriminate the different information contained within the radar measurements.

[0087] The descriptors fragment the information represented in the radar measurement into encoding different geographic areas defined by a pixel area, and whose shape discriminates the presence or absence of a known shape architecture in said area

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] area. The query matrix Q, and possibly that of the keys K, is therefore a matrix of size kxN, with k the number of zones evaluated within the radar measurement, and N, the number of pixels which make up the zone. A very generic solution for defining the areas to be evaluated is to scan the entire measurement following a sliding pattern. As an optional addition, said at least one attention mechanism is: - a self-attention mechanism capable of translating the similarity of different patterns within the same radar measurement or the same time series; or - a specific attention mechanism capable of associating areas of the same radar image with a set of predefined patterns representative of information previously available on the landing environment. In other words, according to this optional supplement, two contexts of application are envisaged, namely: - on the one hand a first generic application context where we assume that we have no information on the expected scene and where the attention mechanism is then a self-attention mechanism capable of translating the similarity of different patterns within the same radar measurement or the same time series, and on the other hand - a second specific application context where, on the contrary, the information relating to the structures supposedly present is assumed to be known, such as the width of the runway, the pattern of the approach ramp, etc., and where the attention mechanism is then a specific attention mechanism capable of associating areas of the same radar image with a set of predefined patterns representative of information previously available on the landing environment. In the first generic application context where it is assumed that no information about the expected scene is available, no specific external information is used. The attention mechanism used then is a so-called "self attention" mechanism, according to which the attention filter reflects the similarity of different patterns within the same radar measurement or the same time series. Such a self-attention mechanism then translates, by definition, into the attribution to the keys K of the same values ​​as the requests. The query matrices Q and the keys K are therefore two identical matrices of dimensions kxN as defined previously. The attention filter: r / qkt \, is then a matrix of size kxk or each softmaxi ~~~ \ ^dk / element of the matrix e'>j constitutes the similarity score between zone i and zone j of the input measurement. The attention mask generated in this way highlights an attention area that groups pixels / areas of pixels together according to their similarities (i.e. repetition of patterns in the measurement).

[0100] In the case of the present specific application of an attention mechanism to a radar measurement, these expressed similarities are therefore manifested in the form of attention masks, which aim to discriminate the relevant characteristic elements present on said measurement. The radar measurement filtered by this or these attention masks (i.e. in the aforementioned case Multi-Head Attention with a multiple number of attention masks), allows the discrimination of the different characteristic elements defined, by the deep learning model of the deep learning tool 12 according to the present invention, as relevant to the purpose of the segmentations requested of it.

[0101] In inference (i.e. real-time application of the previously trained deep learning tool), the attention masks defined by the training of the model advantageously highlight the different characteristic elements of the approach scene visible and relevant on the radar measurement by enhancing the contrast and / or homogenizing an area.

[0102] In other words, the module for implementing at least one self-attention mechanism 14 allows an enhancement of the contrast and / or homogenization of an area to improve the subsequent segmentation.

[0103] In the second specific application context where the information relating to the supposedly present structures is assumed to be known, such as the width of the runway, the pattern of the approach ramp, etc., there is on the contrary exploitation of known information(s) on the approach scene.

[0104] Indeed, the assessment of distances permitted by the use of radar, particularly millimetric radar, allows the search for characteristic elements of the approach scene with codified architecture and dimensions.

[0105] According to an example of use, such a characteristic element corresponds to the approach ramp, and for example a portion of ramp is selected whose presence is expected or even simply a standard ramp pattern, conventionally present in different approach scenes, without being specific to the one expected, projected into the radar field by exploiting the known characteristics of the radar, namely for example the resolution of the distance and circular angular dimension and / or the inertial data of the radar if necessary.

[0106] If, in the generic case relating to the first aforementioned context, we associate the zones within the same image by similarity, we seek this time according to this second context where we use a specific attention mechanism to associate these zones with predefined patterns illustrating these standard architectures supposedly present and known.

[0107] The use of such a specific attention mechanism in this second context where the information relating to the supposedly present structures is assumed to be known is therefore based on the generation of a dictionary of patterns extracted from regulatory architectures expected on an approach scene.

[0108] In this second context, the queries Q remain the multiple zones of the input radar image defined by its descriptors as described previously. The keys K, to which the queries Q associated with this specific attention mechanism will be compared, are the characteristic elements of the dictionary of patterns generated previously. Note that a query and a key both translate a zone of the same surface area and therefore composed of the same number of pixels.

[0109] The query matrix therefore remains, in this second context, a matrix of dimension kx N with k the number of zones selected within the radar image, and N the number of pixels which define the zone, while the key matrix K is a matrix of size lx N with / the number of patterns present in the dictionary. The attention filter is a matrix of size kxl where each element of the matrix represents a similarity score between the zone ï of the radar measurement, and the pattern j of the pattern dictionary, taking into consideration the different possible positions and orientations of the characteristic elements in the radar reference frame, as well as the deformations of the scene which follow. Note that by characteristic elements, we mean the potentially present (and important) infrastructures expected on the approach scene in the landing phase, such as for example the ramp.The patterns themselves are defined to “fit” best to these characteristic elements or at least to a piece of these elements.

[0110] Filtering by such an attention filter thus estimated makes it possible to define an area solely by a combination of its similarities with the patterns of the dictionary, and it is on this interpretation of the radar measurement that the subsequent localization of the different characteristic elements is based as such as implemented by the device according to the present invention and described below.

[0111] It should be noted that, regardless of the context, the self-attention mechanism, like the specific attention mechanism, makes it possible to highlight the relevant elements, to exploit the relative positions of the characteristic elements of the approach scene, which follow classic and immutable patterns, and also allows the recognition of known and / or expected patterns.

[0112] As indicated previously, the deep learning tool 12 also optionally comprises a predetermined number of recursive cells 16 (also called recursive neural network) LSTM (from the English Long Short Terni Memory) convolutional whose architecture, in a known manner, combines the use of an internal memory adapted to the processing of time series and a convolution element suitable for image processing.

[0113] According to the present invention, such convolutional LSTM recursive 16 cells are specifically exploited in the manner described below.

[0114] As illustrated by [Fig. 1], a series of radar measurements composed of several consecutive instants tN 4, tN.3, tN 2, t\ iCt tN along the time axis t, of the same approach constitutes the input of the layers of attention mechanisms, the outputs of which, also in the form of time series, are provided as input to said LSTM recursive cells 16.

[0115] In other words, the LSTM cell associated with the first instant tN 4 of the series extracts the information from this instant, in particular thanks to its convolution element. This information is transmitted to the following LSTM cell associated with the second instant tN.3, in accordance with what is called the internal memory.

[0116] The next LSTM cell associated with the second instant tN 3 processes the measurement associated with the next instant tN.3, and also takes into account the information transmitted from the previous instant tN 4 in its analysis, before itself transmitting the information to the next LSTM cell associated with the second instant tN 2 and so on until the last element of the time series.

[0117] It should be noted that the exploitation of such convolutional LSTM recursive cells is certainly known for the prediction of the next image of a video stream as notably described by Shi, Xingjian, et al. in the article entitled "Convolutional LSTM network: A machine learning approach for precipitation nowcasting." Advances in neural information processing Systems 28 (2015). In this document, the displacement of the different elements present on the stream is learned by such convolutional LSTM recursive cells but used for the purpose of predicting the next position (i.e. tracking).

[0118] According to the present invention, this information is not used for the purpose of predicting the next position but is specifically used for the purpose of locating predetermined characteristic elements of an aircraft landing environment, to ensure the consistency of the location at an instant in relation to previous instants.

[0119] As an optional addition, the acquisition step between each radar measurement, in particular each millimeter measurement, of said time series is greater than or equal to half a second, and in which the total duration of said time series is less than or equal to two seconds.

[0120] Such a step allows a selection of a time series. Indeed, depending on the frequency of the radar used for real-time acquisition during the inference phase, it is not necessarily relevant to use the measurements of all successive instants. With a relatively high radar acquisition frequency, two consecutive measurements may be too similar for it to be relevant to combine the information from different times within a recursive neural network. This is why, according to this optional addition, such a step makes it possible to advantageously favor a series of radar measurements with a relatively high time step of at least half a second, and whose total duration does not exceed two seconds, in particular due to the TTA (Time To Alarm) alarm delay corresponding to the minimum time before notifying an error during the data measurement.

[0121] Finally, according to the embodiment illustrated by [Fig. 1], the output of the predetermined number of 16 recursive LSTM cells is transmitted as input to a convolutional neural network 18. The use of the 16 recursive LSTM cells allows in particular that the “filtering” associated with the last instant tN has taken into account the information associated with the preceding instant tN.i, the information associated with the preceding instant tN.i having itself taken into account the information associated with the preceding instant tN 2, etc., so that all the information associated with the preceding instants is taken into account recursively from one LSTM cell to another.

[0122] By definition, a neural network comprises an ordered succession of layers of neurons, each of which takes its inputs from the outputs of the previous layer. More precisely, each layer comprises neurons taking their inputs from the outputs of the neurons of the previous layer, or from the input variables for the first layer.

[0123] Alternatively, more complex neural network structures can be envisaged with a layer that can be connected to a layer further away than the immediately preceding layer.

[0124] Each neuron is also associated with an operation, i.e. a type of processing, to be carried out by said neuron within the corresponding processing layer.

[0125] Each layer is connected to the other layers by a plurality of synapses. A synaptic weight is associated with each synapse, and each synapse forms a connection between two neurons. It is often a real number, which takes both positive and negative values. In some cases, the synaptic weight is a complex number.

[0126] Each neuron is capable of performing a weighted sum of the value(s) received from the neurons of the previous layer, each value then being multiplied by the respective synaptic weight of each synapse, or link, between said neuron and the neurons of the previous layer, then applying an activation function, typically a non-linear function, to said weighted sum, and delivering at the output of said neuron, in particular to the neurons of the following layer which are connected to it, the value resulting from the application of the activation function. The activation function makes it possible to introduce non-linearity into the processing carried out by each neuron. Examples of activation functions are the sigmoid function, the hyperbolic tangent function, and the Heaviside function.

[0127] As an optional addition, each neuron is also capable of applying, in addition, a multiplicative factor, also called bias, to the output of the activation function, and the value delivered at the output of said neuron is then the product of the bias value and the value from the activation function.

[0128] According to the present invention, the neural network 18 is specifically a convolutional neural network, also sometimes called a convolutional neural network or by the acronym CNN which refers to the English term “Convolutional Neural Networks”.

[0129] By definition, in a convolutional neural network, each neuron in the same layer has exactly the same connection pattern as its neighboring neurons, but at different input positions. The connection pattern is called a convolution kernel or, more often, a "kernel" in reference to the corresponding English term.

[0130] According to the present invention, such a convolutional neural network 18 has a convolutional neural network architecture particularly suited to image processing, or to processing a multi-dimensional signal in general. The convolution is two-dimensional 2D when the radar measurement is reduced beforehand, in pre-processing, to only the distance and circular dimensions, three-dimensional 3D when it is desired to include the Doppler dimension in the learning, and 4D when it is desired to include the elevation dimension in the learning.

[0131] Any known more or less deep convolutional neural network is suitable for use according to the present invention, such as in particular, a segmentation neural network such as the U-Net network can carry out the segmentation task within the deep learning tool 12 which specifically according to the present invention completes it by means of the information processed / extracted by the attention mechanisms 14 and optionally the recursive cells 16, in order to improve the segmentation performance.

[0132] Such a U-Net network having shown good performance in a segmentation context like the U-Net network is notably introduced by Ronneberger, Olaf, Philipp Fischer, and Thomas Brox in the article entitled "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.

[0133] In order to train the deep learning tool 12 to establish decision rules for segmenting the approach scene, as described below in relation to [Fig.2] illustrating the method implemented according to the present invention, supervised learning of the deep learning tool 12 is implemented.

[0134] For this, a learning database composed of input-output pairs is created. The inputs correspond to the radar measurements E, while the outputs are the ground truths represented in the form of annotated images indicating the presence of the characteristic elements of the approach scene (e.g. approach ramp, runway edge lights, etc.).

[0135] As indicated previously, the neural network 18 of the deep learning tool 12 is composed of weights that we seek to optimize. These weights are firstly initialized randomly and we seek to optimize the values ​​of these weights in order to have the best possible prediction. For this purpose, an iterative algorithm is implemented in order to carry out the supervised learning of the weights of this network. The objective is to minimize a cost function 20 which quantifies the difference between the prediction of the model and the ground truth.

[0136] A batch of data composed of a set of several inputs is passed into the deep learning tool 12, thus making it possible to obtain a prediction for each of these inputs. In order to qualify the performance of the network, the cost function 20 is calculated. Then, a step called back-propagation of the gradient of the cost function 20 is implemented. Depending on the error made, the weights of the neural network 18 are more or less strongly corrected. The larger the cost function 20 (i.e. large error between the prediction and the ground truth), the more the weights are modified. These steps are repeated many times until the learning of the neural network 18 converges.

[0137] For example, the cost function 20 used in such a segmentation or classification context is the cross-entropy r,)log(n) + (ir,)iog(iy)This cost function is minimized during learning via the backpropagation process.

[0138] Once trained, the deep learning tool 12 is configured to be applied in the inference phase to a current radar image obtained, from at least one radar measurement, in particular at least one millimeter radar measurement, current, in the approach phase before landing of the aircraft, so that, via such a trained deep learning tool 12, the device 10 according to the present invention is able to implement a segmentation of said current radar image by determining whether or not a pixel of said current radar image belongs to one of the predetermined characteristic elements of the landing environment (in particular via the determination of a probability of belonging), each of said characteristic elements corresponding to a class of said segmentation, in particular in the case of multi-class segmentation making it possible to isolate the detection of each element to evaluation or visualization purposes. Such characteristic elements of the segmentation result R_S, correspond in particular according to the application of the present invention to the ramp R, the track P, the set L of lamps of the ramp, the axis A of the track, the threshold S of the track, etc.

[0139] In other words, the output of the convolutional neural network 18 is therefore presented in the form of a tensor of dimension Distance X Azimuth XN b Classes. Each pixel or voxel is assigned a probability of belonging to each of the classes considered, in the form of a pseudo-probability index between 0 and 1, and from these indices and their analyses, a pixel will be assigned or not to a class.

[0140] It should be noted that according to another embodiment, not shown, the order of chaining of the constituent modules of the deep learning tool 12 is different, the optional module for implementing a predetermined number of recursive LSTM cells 16, being connected to the module for implementing at least one attention mechanism 14, connected to a module for implementing a convolutional neural network 18.

[0141] In other words, according to this alternative embodiment to that of [Fig.l] and not shown, said LSTM recursive cells receive as input said radar measurements, in particular millimeter radar measurements, grouped in time series, the acquisition frequency of each radar measurement, in particular millimeter, of a time series being predetermined, the output of said LSTM recursive cells being provided as input to said at least one attention mechanism whose output is provided as input to said convolutional neural network. Thus, according to this embodiment not shown, the time series is processed as is by the convolutional LTSM cells 16, the output of which is then processed by the attention mechanisms 14.

[0142] It should be noted that the embodiment of [Fig.l] and this alternative embodiment not shown have each experimentally demonstrated their effectiveness in improving the segmentation of radar images of the approach scene, the most effective embodiment however remaining the embodiment illustrated by [Fig.l].

[0143] The electronic device 10 according to the present invention further comprises a localization module 22 configured to implement the localization of at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar using the result of said segmentation.

[0144] In other words, such a location module 22 is configured to deduce the position information of at least one of the characteristic elements using the segmentation result R_S provided by the deep learning tool 12. Such a segmentation result R_S corresponding in particular to a probability of belonging of a pixel of said current radar image to one of the predetermined characteristic elements of the landing environment, each of said elements ca characteristics corresponding to a class of said segmentation, in particular in the case of multi-class segmentation allowing the detection of each element to be isolated for evaluation or visualization purposes.

[0145] As an optional addition, said module 22 for locating at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar is configured to implement at least the location of the landing runway and comprises, in a manner not shown, a post-processing tool configured to implement

[0146] - a first post-processing comprising:

[0147] - the determination, for each predetermined characteristic element, of at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance 26 and circular 28, the circular of a point corresponding to the angle formed from the origin of said reference frame and the radar sighting axis to the projection of said point in the plane (0, x, y) and oriented positive following the rotation around the axis (0, z); and

[0148] - determining the orientation 30 of the axis of the landing runway in the reference frame of said radar, by:

[0149] - correlation between the result of said segmentation and a known contour of said track; and / or

[0150] - identification of track edges by determining a plurality of points of interest whose contour contrasts in said current radar image by being representative of a track rectangle whose lines are obtained by linear regression from said plurality of points, or

[0151] or

[0152] - a second post-processing comprising the determination, by neural network, of said at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance and circular, and of said orientation of the axis of the landing runway in the reference frame of said radar.

[0153] It should be noted that the application of the device 10 for localization purposes, and in particular of the landing strip, clearly encourages favoring segmentation detection tools (i.e. providing an image segmentation result) over detection tools as such which would be adapted to “point” objects but not to large areas such as the concrete on which the landing strip is located and which constitutes a common element in almost all aircraft approach scenes.

[0154] More precisely, from the probabilities of the segmentation result R_S, information is extracted / deduced which notably allows the superposition of a synthetic track (by estimating in particular the rectangle most consistent with said probabilities per pixel) allowing to quantify a location, the orientation of the runway in relation to the radar line of sight, the axis of the approach ramp when it is present, the location of the runway threshold, the location of the intersections between the longitudinal and transverse light ramps, etc.

[0155] The location of the aircraft relative to the runway is established by two elements, corresponding firstly to a longitudinal reference point located on the axis of the runway: the middle of the physical threshold of the runway (in the sense of the end of its covering), generally coinciding with the vertical line of the LTP (from the English Landing Threshold Point)•

[0156] This point is characterized by its coordinates in the radar reference frame in circular distance pi5 a; and possibly its coordinate in the Doppler dimension.

[0157] Alternatively, subject to knowing the distance from this(these) point(s) to the middle of the standardized runway threshold corresponding to the LTP, this point may be the middle of an “offset” threshold (in the aerodrome / airport sense) or an intersection of approach lights.

[0158] The other element defining the location of the aircraft is the orientation y 30 of the runway in the radar reference frame. This orientation 30 is suitable for being estimated by the location module 22 as mentioned above by correlation, and / or from a plurality of points locating the edges of the runway (identified by the contrasts of the rectangle of the runway) and / or the runway centerline lights and / or the longitudinal approach lights, or by learning by artificial intelligence using a neural network (independently of the attitudes of the aircraft).

[0159] In particular, when the localization module 22 implements a neural network, such a neural network is capable of implementing a linear regression corresponding for example according to a basic form to a multi-layer perceptron which makes it possible to efficiently and simply estimate the distance 26, the circular 28, and the orientation 30 of the axis from the results of the segmentation.

[0160] In particular, such a neural network of the localization module 22 comprises a fully connected layer of neurons which is a layer in which the neurons of said layer are each connected to all the neurons of the previous layer. Such a type of layer is more often referred to as “fully connected”, and sometimes referred to as a “dense layer”.

[0161] Such a neural network of the localization module 22 is previously trained, said training also being based on an iterative calculation aimed at minimizing a cost function 24 quantifying the difference between the neural network prediction of the localization module 22 and the associated ground truth (i.e. the localization as such according to the ground truth). For example, this cost function aims to minimize the mean square error.

[0162] These analyses and post-processing of the segmentation provide several possible applications adapted to several contexts, depending on the information made available, such as the presence or absence of an approach ramp, the width of the runway, etc.

[0163] More precisely, the determination of the orientation of the runway is based on the assumptions of a contrast, measured by the radar by differentiation of the amplitudes of the reflections between the surface and the surroundings of the runway, and of a known runway contour such as for example: a rectangle with or without access ramps or a U-turn circle. The location of the runway rectangle is carried out in a Cartesian reference frame, preferably that of the radar, and takes into consideration the width of the runway.

[0164] For this, the radar measurements are scanned along the distance and circular dimensions, and according to a first post-processing, a correlation between the result of the track segmentation and the hypothetical track is carried out.

[0165] The result of this 2D two-dimensional correlation makes it possible to estimate the most probable location of the track using the maximum correlation. In the event that the track width information is not known, the quantile is used to define the contours of the track by estimating a higher probability of false detection at the ends of the detection. The width information can also be used to evaluate the consistency of the detection. Finally, the path of the circular dimension can be restricted to the vicinity of “Skeleton” type information providing an approximation of this angle.

[0166] The analysis of the detected lamps of the ramp also makes it possible to evaluate this consistency. The estimation of the centroid of the different lamps makes it possible to verify several criteria validating the standard architecture of a ramp. The alignment of the detected lamps of the ramp in the geographical reference can be verified, thus eliminating inconsistent detections and / or using the axis of this detected ramp as information to be used by the aircraft. The regularity of the spacing of the lamps can also be verified by assuming the distance separating two lamps is known or unknown.

[0167] The aforementioned examples of possible analyses and post-processing on the segmentation results thus allow validation by consistency with the known truth of the field or the search for elements with known characteristics. For example, advantageously, the arrangement of the synthetic track in accordance with the predictions can take into account the width of the track if this is assumed to be known to the user.

[0168] The electronic device 10 for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment therefore comprises an obtaining module, a deep learning tool 12 comprising a module for implementing at least one attention mechanism 14 connected to an optional module for implementing a pre-defined number of determined from 16 recursive LSTM cells, connected to a module for implementing a convolutional neural network 18, said deep learning tool being connected to a localization module 22, as shown in [Fig.l].

[0169] In the example of [Fig.l], in a manner not shown, the electronic device 10 for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment comprises an information processing unit formed for example of a memory and a processor associated with the memory.

[0170] In the example of [Fig.l], the obtaining module, the module for implementing at least one attention mechanism, the module for implementing a convolutional neural network and the localization module, as well as, as an optional addition, the module for implementing a predetermined number of LSTM recursive cells, are each implemented in the form of software, or a software brick, executable by the processor. The memory of the electronic device for segmenting image(s) and localizing predetermined characteristic elements of an aircraft landing environment is then capable of storing obtaining software, software for implementing at least one attention mechanism, software for implementing a convolutional neural network and localization software, as well as, as an optional addition, software for implementing a predetermined number of LSTM recursive cells.The processor is then able to execute each of the software among the obtaining software, the software for implementing at least one attention mechanism, the software for implementing a convolutional neural network and the localization software, as well as, as an optional addition, the software for implementing a predetermined number of LSTM recursive cells.

[0171] In a variant not shown, the obtaining module, the module for implementing at least one attention mechanism, the module for implementing a convolutional neural network, the localization module, as well as, as an optional addition, the module for implementing a predetermined number of 16 recursive LSTM cells, are each produced in the form of a programmable logic component, such as an FPGA (Field Programmable Gate Array) or an integrated circuit, such as an ASIC (Application Specific Integrated Circuit).

[0172] When the electronic device for segmenting hang(s) and locating predetermined characteristic elements of an aircraft landing environment therefore comprises a module for obtaining it is produced in the form of one or more software programs, that is to say in the form of a computer program, also called a computer program product, it is furthermore capable of being recorded on a medium, not shown, readable by a computer. The computer-readable medium is for example a medium capable of storing electronic instructions and of being coupled to a bus of a computer system. For example, the readable medium is an optical disc, a magneto-optical disc, a ROM memory, a RAM memory, any type of non-volatile memory (for example FLASH or NVRAM) or a magnetic card. A computer program including software instructions is then stored on the readable medium.

[0173] An example embodiment of the operation of such a device 10 according to the present invention is described below in relation to [Fig.2].

[0174] More precisely, the method 40 for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from radar measurements, in particular millimeter radar measurements, with M dimensions, with 2 < M < 4, said aircraft carrying at least one radar, in particular a millimeter radar, firstly comprises a first optional step 42 of preprocessing each radar measurement, in particular each millimeter measurement.

[0175] Indeed, before launching a learning process on the radar data, a preprocessing step is optionally and advantageously implemented. Indeed, beyond a certain distance, the signal-to-noise ratio is too low and the radar measurements are generally therefore not usable. In order not to disrupt the learning of the neural network 18 of the deep learning tool 12 as described previously in relation to [Fig.l], the radar measurements are initially preprocessed in order to retain only the measurements whose signal-to-noise ratio is sufficient. In practice, only radar measurements having a distance of less than 2200 m are retained.

[0176] In other words, according to an optional aspect, such a preprocessing step 42 comprises a filtering 44 retaining only the radar measurements, in particular the millimeter radar measurements, having a distance less than 2200m, and / or a reduction 46 of the dimension of each radar measurement to a two-dimensional or three-dimensional radar measurement.

[0177] Then, the method 40 comprises a phase 48 of supervised learning of a deep learning tool.

[0178] Such a supervised learning phase 48 firstly comprises a step 50 of obtaining a labeled learning data set comprising images, obtained from radar measurements, in particular millimeter radar measurements, with M dimensions acquired beforehand during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar.

[0179] In addition, such a phase 48 of supervised learning of a learning tool deep learning comprises a step 52 of training said deep learning tool on said labeled training data set, said deep learning tool combining, as described previously, at least one attention mechanism and a convolutional neural network, said training being based on an iterative calculation aimed at minimizing a cost function quantifying the difference between the prediction of said deep learning tool and the associated ground truth.

[0180] At the end of said learning phase 48, a trained deep learning tool 56 is therefore obtained.

[0181] Then, the method 40 comprises an inference phase 58, applying said trained deep learning tool to a current radar image obtained, from at least one radar measurement, in particular a current millimeter measurement, in the approach phase before landing of the aircraft.

[0182] Such an inference phase 58 firstly comprises a step 60 of segmentation of said current radar image by said trained deep learning tool 56 by determining whether or not a pixel of said current radar image belongs to one of the predetermined characteristic elements of the landing environment, each of said characteristic elements corresponding to a class of said segmentation.

[0183] Such an inference phase 58 also comprises a step 62 of locating at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar using the result of said segmentation.

[0184] Several optional alternatives to the above steps are described below.

[0185] We first describe in more detail the step 50 of obtaining said learning data set (i.e. database) of the learning phase 48.

[0186] Indeed, according to the application of the present invention to the field of aircraft navigation, the learning database is made up of radar measurements acquired during landing procedures.

[0187] In order to carry out generalized learning which allows correct prediction on data not represented in the learning database, the latter must be as exhaustive as possible, with a significant diversity of scenes represented, as well as a significant diversity of acquisition contexts, particularly in terms of meteorological conditions, the aircraft's line of sight, etc.

[0188] In the context of observing an aerodrome, the presence of the characteristic elements of the approach scene, such as the approach ramp, the runway edge lights, etc. and the arrangement of these elements are codified as indicated previously. Even if they present a certain variability from one landing scene to another, such as for example a more or less wide spacing between the lights of an approach ramp, it remains nonetheless that these elements are characteristic and representative of a landing scene. This therefore makes it possible to envisage a strategy supervised learning for runway detection using radar measurements.

[0189] According to the present invention, the supervised learning is carried out on a database composed of radar measurements acquired during various landing procedures. In this database are represented all the categories of approaches as defined in the document of the International Civil Aviation Organization (ICAO), Annex 6, paragraph 1, namely categories I, II, IIIA, IIIB, IIIC, as well as the different existing models of approach ramp (ALSF-1, ALSF-2, (with ALSF from the English Approach Lighting System with Sequence Flashing Lights), MALSF (with ALSF from the English Medium Intensity Approach Lighting System with Sequence Flashing Lights), etc.).

[0190] Furthermore, as indicated previously, to carry out supervised learning of a neural network, it is necessary to use a learning database containing the ground truth.

[0191] In a segmentation context, the ground truth is presented in the form of a map of at least two dimensions, namely the distance dimension and the circular dimension, and optionally in addition the Doppler dimension and the elevation dimension where each pixel is labeled as belonging to a class, such as for example; the concrete runway, the approach ramp, the approach threshold, etc.

[0192] In order to create this ground truth, the geographical coordinates of each of these elements of the approach scene are recovered using survey tools such as Google Maps®, Geoportail®. Then, thanks to the characteristics of the radar signal used as well as the inertial data of the aircraft, it is possible to project the position of the different characteristic elements into the radar reference frame and thus produce the ground truths.

[0193] As mentioned previously, the classes represent the different elements of a scene whose location could be relevant to the pilot. Without being exhaustive, this concerns the approach ramp, the runway edge lights, the concrete runway, etc.

[0194] Advantageously, as the runway edge lights are generally located on the concrete runway, the neural network used optionally adopts a multi-class approach, in the form of a multi-channel label, thus allowing a pixel to belong to several classes.

[0195] In other words, as an optional addition, within step 50 of obtaining learning phase 48, said deep learning tool 12 is configured to apply a multi-class approach by first using a sub-step 64 of multiple channel labeling L_C_M of said learning data set according to which a pixel is capable of belonging to several classes of said characterizing elements. predetermined characteristics.

[0196] The training step 52 of the learning phase 48 and / or, in a manner not shown, the segmentation step 60 of the inference phase 58, correspond respectively to the iterative implementation 66 of the deep learning tool during the training step 52 of the learning phase 48, and to the implementation of the trained deep learning tool during the segmentation step 60 of the inference phase 58.

[0197] More precisely, each of the implementations of the deep learning tool to be trained (in learning phase 48) and trained (in inference phase 58) comprises a step 68 of implementing at least one attention mechanism optionally corresponding to a self-attention mechanism MSA or a specific attention mechanism MAS.

[0198] In addition, as an optional addition, during step 68 of implementing at least one attention mechanism, said at least one attention mechanism comprises a number of masks M greater than or equal to the number of classes of predetermined characteristic elements of the landing environment.

[0199] In addition, as an optional addition, during step 68 of implementing at least one attention mechanism, said at least one attention mechanism comprises the coding C of at least one descriptor of the relative position, in the spatial or temporal domain, of said predetermined characteristic elements with respect to each other within said landing environment.

[0200] Optionally, as indicated previously in relation to [Fig.l], the implementation 66 of the deep learning tool comprises a step 72 of using a predetermined number of LSTM recursive cells in order to implement an LST localization exploiting time series.

[0201] Furthermore, as indicated previously in relation to [Fig.l], the implementation 66 of the deep learning tool comprises a step 70 of implementing a convolutional neural network RNC.

[0202] With regard to the inference phase 58, the segmentation step 60 has been previously described and consists of applying the trained deep learning tool 56 provided by the learning phase 48 to a current radar image.

[0203] Step 62 of locating at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar using the result of said segmentation, optionally comprises as previously indicated in relation to [Fig.l]:

[0204] - a step 76 of applying a first post-processing comprising two sub-processes steps 76 and 78 of determining respectively at least one point corresponding to the middle of the runway threshold S and the orientation y of the runway axis;

[0205] or

[0206] - a step 80 of applying a second post-treatment comprising the determination mination, by neural network, of said at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance and circular, and of said orientation of the axis of the landing runway in the reference frame of said radar.

[0207] Furthermore, according to an optional aspect, not shown, the method further comprises a step of providing said segmentation result R_S and / or said location to a pilot of said aircraft and / or to automatic pilot equipment of said aircraft.

[0208] In other words, according to this optional aspect, the segmentation result R_S and / or said location are suitable for being provided to an automatic pilot or to a human pilot, in particular in the aircraft reference frame which corresponds to that of the radar to within a simple fixed translation. For example, such provision is implemented within the framework of a visual function for assisting approaches and landings, such as the superposition of the trapezoid of the runway using an enhanced flight vision system, EFVS (from the English Enhanced Flight Vision System) or a synthetic vision system SVS.

[0209] Those skilled in the art will understand that the invention is not limited to the embodiments described, nor to the particular examples of the description, the embodiments and variants mentioned above being suitable for being combined with each other to generate new embodiments of the invention.

[0210] The present invention thus makes it possible to participate in the localization of different elements characterizing an approach scene by using a segmentation model associated with the aforementioned deep learning tool.

[0211] Such a deep learning tool combines at least the convolution layers, which are here applied specifically to radar images, the radar being on board the aircraft, attention mechanisms which make it possible to highlight the relevant elements, to exploit the relative positions of the elements of the approach scene, which follow classic and immutable patterns while also allowing the recognition of known and / or expected patterns, and optionally a recursive architecture making it possible to extract and interpret the information from different time instants.

[0212] Such efficient segmentation and precise localization provides information to the pilot (human or automatic), making it possible to facilitate or ensure the landing maneuver.

Claims

Claims

1. Method (40) for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from M-dimensional radar measurements, with 2 < M < 4, said aircraft carrying at least one radar, said method comprising: - a phase (48) of supervised learning of a deep learning tool comprising: - obtaining (50) a labeled learning data set comprising images, obtained from M-dimensional radar measurements acquired beforehand during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar; - training (52) said deep learning tool on said labeled training data set, said deep learning tool combining at least one attention mechanism and a convolutional neural network, said training being based on an iterative calculation aimed at minimizing a cost function quantifying the difference between the prediction of said deep learning tool and the associated ground truth, - an inference phase (58), applying said trained deep learning tool to a current radar image obtained, from at least one current radar measurement, in the approach phase before landing of the aircraft, comprising: - the segmentation (60) of said current radar image by said deep learning tool trained by determining whether or not a pixel of said current radar image belongs to one of the predetermined characteristic elements of the landing environment, each of said characteristic elements corresponding to a class of said segmentation; - the localization (62) of at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar using the result of said segmentation.

2. The method of claim 1, wherein said deep learning tool further comprises a predetermined number of LSTM recursive cells, said LSTM recursive cells receiving as input said measurements radar grouped in time series, the acquisition frequency of each radar measurement of a time series being predetermined, the output of said LSTM recursive cells being provided as input to said at least one attention mechanism whose output is provided as input to said convolutional neural network or said at least one attention mechanism receiving as input said radar measurements grouped in time series, the acquisition frequency of each radar measurement of a time series being predetermined, the associated outputs of said at least one attention mechanism being provided as input to said recursive cells whose respective outputs are provided as input to said convolutional neural network.

3. The method of claim 2, wherein the acquisition step between each radar measurement of said time series is greater than or equal to half a second, and wherein the total duration of said time series is less than or equal to two seconds.

4. Method according to any one of the preceding claims, wherein said at least one attention mechanism comprises a number of masks greater than or equal to the number of classes of predetermined characteristic elements of the landing environment.

5. A method according to any preceding claim, wherein said at least one attention mechanism comprises encoding at least one descriptor of the relative position, in the spatial or temporal domain, of said predetermined characteristic elements with respect to each other within said landing environment.

6. Method according to any one of the preceding claims, in which said at least one attention mechanism is: - a self-attention mechanism capable of translating the similarity of different patterns within the same radar measurement or the same time series; or - a specific attention mechanism capable of associating zones of the same radar image with a set of predefined patterns representative of information previously available on the landing environment.

7. A method according to any one of the preceding claims, wherein said deep learning tool is configured to apply a multi-class approach by first using a multi-channel labeling of said training dataset according to which a pixel is capable of belonging to several classes of said characterizing elements. predetermined characteristics.

8. Method according to any one of the preceding claims, further comprising a step of providing said segmentation result and / or said location to a pilot of said aircraft and / or to automatic pilot equipment of said aircraft, and / or wherein said location of at least one of the predetermined characteristic elements of the landing environment in the reference frame of said radar corresponds at least to the location of the landing runway and comprises: - a first post-processing including: - the determination of at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance and circular, the circular of a point corresponding to the angle formed from the origin of said reference frame and the line of sight of the radar to the projection of said point in the plane (0, x, y) and oriented positive following the rotation around the axis (0, z); and - determining the orientation of the axis of the landing runway in the reference frame of said radar, by: - correlation between the result of said segmentation and a known contour of said track; and / or - identification of runway edges by determining a plurality of points of interest whose contour contrasts in said current radar image by being representative of a runway rectangle whose straight lines are obtained by linear regression from said plurality of points, or - a second post-processing comprising the determination, by neural network, of said at least one point corresponding to the middle of the runway threshold, defined by its coordinates in the reference frame of said radar in distance and circular, and of said orientation of the axis of the landing runway in the reference frame of said radar.

9. A method according to any preceding claim, wherein each radar measurement is preprocessed before use during the learning and / or inference phase, the preprocessing of each radar measurement comprising: - filtering retaining only radar measurements with a distance less than 2200m, and / or - a reduction in the dimension of each radar measurement to a two-dimensional or three-dimensional radar measurement.

10. Electronic device (10) for segmenting image(s) and locating predetermined characteristic elements of an aircraft landing environment within image(s) obtained from M-dimensional radar measurements, with 2 < M < 4, said aircraft carrying at least one radar, said device being configured to implement: - a supervised learning phase of a deep learning tool (12) comprising: - obtaining a labeled training data set comprising images, obtained from M-dimensional radar measurements acquired beforehand during past landings of the aircraft, associated with a ground truth, corresponding to annotated images where the predetermined characteristic elements of the landing environment of an aircraft are located in the reference frame of said radar; - training said deep learning tool (12) on said labeled learning data set, said deep learning tool combining at least one attention mechanism (14) and a convolutional neural network (18), said training being based on an iterative calculation aimed at minimizing a cost function quantifying the difference between the prediction of said deep learning tool (12) and the associated ground truth, - an inference phase, applying said trained deep learning tool (12) to a current radar image obtained, from at least one current radar measurement, in the approach phase before landing of the aircraft, comprising: - the segmentation of said current radar image by said deep learning tool (12) trained by determining whether or not a pixel of said current radar image belongs to one of the predetermined characteristic elements of the landing environment, each of said characteristic elements corresponding to a class of said segmentation; - the location of the predetermined characteristic elements of the landing environment in the reference frame of said radar using the result of said segmentation.

Images