Method and device for tracking satellites in an image sequence

Abstract

The present invention relates to a method and device (APP) for tracking satellites in a sequence of images (IMG_SEQ). In particular, the method comprises: a step of obtaining a set of measurements (SIN) associated with spatial objects observed in the images of the sequence (IMG_SEQ), each measurement comprising a position of an object in an image and a temporal index of the image; a step of detecting satellite trajectories (TRJ) in the sequence of images (IMG_SEQ), by a detector (DET) comprising at least one neural network and using all or part of the set of measurements obtained (SIN); and a step of classifying the detected satellite trajectories (TRJ), by a classifier (CLA) comprising at least one other neural network. Figure for abstract: Fig. 2

Description

Title of the invention: Method and device for tracking satellites in an image sequence Technical field

[0001] The present invention relates to the fields of image analysis and object tracking. In particular, the present invention relates to a method and device for tracking satellites in an image sequence, as well as an associated monitoring system, computer program and information medium. The present invention finds a particularly advantageous, although in no way limiting, application for the implementation of space surveillance systems. Prior art

[0002] The invention falls within the particular context of outer space surveillance systems. Such systems aim to locate satellites in orbit and provide their respective trajectories. To this end, these systems use optical sensors to capture images of satellites orbiting the Earth. However, these images include satellites, stars, and noise (e.g., hot pixels).

[0003] Also, we are interested below in surveillance systems capable of extracting the trajectories of the different satellites observed in a sequence of images. In other words, these systems aim to provide as output a set of measurements associated with the same satellite to describe its trajectory over time, that is to say a set of three-dimensional coordinates associated with this satellite (spatial positions x and y, and temporal index).

[0004] In the current state of the art, there are analytical solutions for tracking satellites within a sequence of images. These analytical solutions exploit the distances between the measurements associated with the observed space objects to extract the trajectories of the satellites and detect the different types of trajectories (e.g. geostationary, moving).

[0005] However, existing satellite tracking solutions have a number of drawbacks, including the following. On the one hand, these analytical solutions involve high implementation complexity, which requires the manual configuration of a multitude of parameters and significant use of computing resources (i.e., significant execution times). On the other hand, the reliability of these solutions, in terms of correctly detected trajectories, is not fully satisfactory.

[0006] There is therefore a need for a solution to track sa tellites in an image sequence reliably and with minimal implementation complexity. Statement of the invention

[0007] The present invention aims to remedy all or part of the drawbacks of the prior art, in particular those set out above.

[0008] According to one aspect of the invention, there is provided a method of tracking satellites in a sequence of images, the method comprising: - a step of obtaining a set of measurements associated with spatial objects observed in the images of the sequence, each measurement comprising a position of an object in an image and a temporal index of the image; - a step of detecting satellite trajectories in the sequence of images, by a detector comprising at least one neural network and using all or part of the set of measurements obtained; and - a step of classifying the detected satellite trajectories, by a classifier comprising at least one other neural network.

[0009] The present invention makes it possible to extract the respective trajectories of the satellites observed in the image sequence. In fact, the present invention makes it possible to associate the measurements belonging to the same satellite to describe its trajectory in the image sequence. It is important to note that the present invention makes it possible not only to track satellites in geostationary orbit, but also satellites in medium or low Earth orbit.

[0010] In addition, the present invention makes it possible to classify the detected trajectories. In particular, the class assigned to each satellite trajectory may indicate the type of orbit of the satellite. For example, the list of assignable classes may include: fixed geostationary trajectory; drifting geostationary trajectory; scrolling type trajectory (i.e. satellite in medium or low Earth orbit moving at high speed); and trailing type trajectory (i.e. satellite in low Earth orbit moving at very high speed).

[0011] Compared to existing analytical solutions, the present invention makes it possible to improve the reliability of tracking observed satellites within an image sequence. It also has reduced implementation complexity (i.e. reduced execution times, less configuration). Advantageously, the present invention does not require a priori knowledge of the orbits to track satellites in an image sequence.

[0012] Indeed, the use of one or more neural networks to implement the detector makes it possible to obtain reliable detection of trajectories, in terms of correctly detected trajectories. The neural network(s) are trained to automatically from training data (i.e. reference image sequences) to optimize the reliability of satellite trajectory detection. This allows, in particular, to obtain significantly improved performance compared to analytical solutions. In addition, the use of one or more neural networks makes it possible to significantly reduce the necessary computing resources, particularly in terms of execution time.

[0013] The same applies to the classifier comprising one or more neural networks, which makes it possible to improve the reliability of classification of trajectories (i.e. correctly identified trajectories). This also makes it possible to reduce the necessary computing resources (i.e. execution time). The joint use of one or more neural networks to detect satellite trajectories, and one or more other neural networks to classify these trajectories makes it possible to obtain reliable tracking of the satellites observed in the image sequence.

[0014] Thus, the present invention makes it possible to track satellites in an image sequence reliably and with minimal implementation complexity.

[0015] According to one embodiment, the detector comprises at least one convolutional neural network, and determines frames each encompassing (surrounding) a satellite trajectory in the images of the sequence. In addition, according to this embodiment, the classifier comprises at least one classification (i.e. segmentation) neural network, in particular at least one PointNet type neural network. And, for each frame encompassing a satellite trajectory, the classifier determines (i.e. associates) the measurements located in the frame belonging to this trajectory, and assigns a class to this trajectory.

[0016] By "convolutional neural network", we refer here to a neural network comprising one or more convolution layers. And, by "PointNet neural network", we refer to a neural network dedicated to processing point clouds, and in particular to classification and segmentation applications. PointNet networks and their implementation are for example described in the following document: Qi et al., "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation", 2017.

[0017] This implementation mode proposes to use one or more convolutional neural networks to detect satellite trajectories, and to use one or more PointNet type neural networks to classify (segment) these trajectories.

[0018] The convolutional neural network(s) allow the detector to exploit the spatio-temporal characteristics of the image sequence to detect the satellite trajectories. More precisely, it exploits the spatio-temporal characteristics of the image sequence to determine bounding boxes (i.e., “bounding boxes”) for the satellite trajectories. The convolutional neural network(s) thus allow reliable detection of the trajectories of the different satellites observed in the image sequence.

[0019] Furthermore, the use of one or more PointNet neural networks to implement the classifier is particularly advantageous. Indeed, the PointNet network is optimized for classifying and segmenting point clouds. Here, the PointNet network processes separately each of the frames encompassing the satellite trajectories. For each frame encompassing a trajectory, it takes as input all the measurements located inside the bounding frame in the image sequence. Thus, these PointNet networks make it possible to reliably separate the measurements belonging to the trajectory from those not belonging to the trajectory, and to assign a class to this trajectory.

[0020] Also, this mode of implementation makes it possible to reliably extract the respective trajectories of the different satellites observed in the sequence of images and to classify them.

[0021] According to one embodiment, the detector comprises at least one Transformer-type neural network, and associates the measurements belonging to the same satellite trajectory in the images of the sequence. Furthermore, according to this embodiment, the classifier comprises at least one classification network and assigns a class to each of the satellite trajectories.

[0022] Here we refer by "Transformer-type neural network" to a neural network exploiting an attention mechanism. Transformer-type neural networks and their implementation are for example described in the following document: A. Vaswani et al., "Attention Is Ail You Need", 2017.

[0023] It should be emphasized that Transformer type networks are optimized to process sequential data and allow the temporal characteristics of the input data to be exploited. In the context of the invention, the use of a Transformer type network allows the detector to exploit the temporal characteristics of the set of measurements to extract the respective trajectories of the satellites observed in the sequence of images.

[0024] This mode of implementation is particularly advantageous in that it makes it possible to improve the reliability of the tracking of the satellites observed in a sequence of images. In addition, the use of a Transformer type network makes it possible to reduce the computing resources (i.e. the execution time) necessary for the implementation of the proposed solution.

[0025] According to one embodiment, the detector determines latent vectors representative of said measurements using said at least one Transformer-type neural network; and the detector associates measurements with the same satellite trajectory as a function of distances between the latent vectors.

[0026] This implementation mode proposes to use the Transformer network to represent in a latent space the measurements associated with the spatial objects observed in the image sequence. For each of the measurements provided as input, the Transformer neural network determines a latent vector.

[0027] Note that the Transformer-type neural network makes it possible to learn, on the basis of training data, a latent space in which the measurements belonging to the same trajectory are close. This network thus makes it possible to represent the measurements in a latent space so as to minimize the distance between the latent vectors of measurements belonging to the same trajectory, and to maximize the distances between the latent vectors of measurements belonging to different trajectories.

[0028] This mode of implementation therefore makes it possible, by using the distances between the latent vectors, to reliably associate (i.e. to group) the measurements belonging to the same trajectory.

[0029] According to one embodiment, the detector comprises a plurality of neural networks respectively dedicated to the detection of satellite trajectories of different speeds; and / or the classifier comprises a plurality of neural networks respectively dedicated to the classification of satellite trajectories of different speeds.

[0030] This implementation mode makes it possible to adapt the detection and / or classification of trajectories according to different types of trajectories (e.g. fixed or drifting geostationary orbits, medium and low Earth orbits). This advantageously makes it possible to improve the reliability of the proposed solution.

[0031] According to one embodiment, the method comprises a step of filtering the set of measurements to eliminate the measurements associated with (i.e. originating from) the stars, and in which the detector uses only the filtered set of measurements to carry out the step of detecting satellite trajectories in the sequence of images.

[0032] This implementation mode makes it possible to eliminate the stars present in the image sequence. Typically, the stars are in large numbers compared to the satellites in an image sequence, which can interfere with the detection of satellite trajectories. Also, this implementation mode contributes to improving the reliability of the proposed solution for tracking satellites in an image sequence. Furthermore, and advantageously, the proposed solution for eliminating measurements corresponding to stars does not require a priori knowledge of their positions.

[0033] According to one embodiment, the step of filtering the set of measurements to eliminate the measurements associated with stars comprises: - a determination of star movement information from measurements whose brightness is greater than a threshold in the images of the sequence; and - a step of elimination (i.e. deletion) of the measures considered as associated with stars from the determined displacement information.

[0034] This mode of implementation is advantageous in that it allows stars to be reliably eliminated in the images of the sequence.

[0035] In fact, the inventors have observed that the majority of the brightest measurements in a sequence of images are stars. Also, it is proposed here to exploit the brightest measurements to determine the displacement information of the stars (e.g. speed of displacement of the stars per image, or rotation angle of the stars per image). Then, this displacement information is used to determine which measurements correspond to stars, and to eliminate these.

[0036] According to one embodiment, the method comprises: - one or more steps of merging (i.e. combining) satellite trajectories to group together the measurements associated with several detected satellite trajectories and belonging to the same satellite; and / or - a step of filtering the satellite trajectories so that a said satellite trajectory includes only one measurement per image of the sequence.

[0037] This implementation mode proposes to carry out different post-processing steps on the satellite trajectories detected by the proposed solution. It thus contributes to improving the reliability of the proposed satellite tracking solution.

[0038] On the one hand, it is proposed to merge (i.e. combine) the trajectories belonging to the same satellite. In other words, if measurements belonging to the same satellite have been associated with a plurality of trajectories, then these measurements are grouped within the same trajectory.

[0039] Furthermore, it is also proposed to filter the different detected satellite trajectories so that each trajectory only includes one measurement per image of the sequence.

[0040] According to another aspect of the invention, there is provided a device for tracking satellites in a sequence of images, the device comprising: - an obtaining module configured to obtain a set of measurements associated with spatial objects observed in the images of the sequence, each measurement comprising a position of an object in an image and a temporal index of the image; - a detector configured to detect satellite trajectories in the image sequence, the detector comprising at least one neural network and using all or part of the set of measurements obtained; and - a classifier configured to classify detected satellite trajectories, the classifier comprising at least one other neural network.

[0041] The satellite tracking device may be configured to carry out any of the modes of implementing a method according to the invention. In particular, for each step of a method according to the invention, the device may comprise a module configured to implement this step.

[0042] The term module can correspond to a software component as well as to a hardware component or a set of hardware and software components. A software component corresponds to one or more computer programs or subroutines, or more generally to any element of a program capable of implementing a function or a set of functions as described for the modules concerned. In the same way, a hardware component corresponds to any element of a hardware assembly capable of implementing a function or a set of functions for the module concerned.

[0043] According to one aspect of the invention, there is provided a surveillance system comprising: a satellite tracking device according to the invention; and a device for acquiring a sequence of images.

[0044] According to one aspect of the invention, there is provided a computer program comprising instructions for implementing the steps of a method according to the invention, when the computer program is executed by at least one processor or computer.

[0045] The computer program may be formed of one or more sub-parts stored in the same memory or in separate memories. The program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0046] According to one aspect of the invention, there is provided a computer-readable information medium comprising a computer program according to the invention.

[0047] The information carrier may be any entity or device capable of storing the program. For example, the carrier may comprise a storage means, such as a non-volatile memory or ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk. Furthermore, the storage medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by a telecommunications network or by a computer network or by other means. The program according to the invention may in particular be downloaded onto a computer network. Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.

[0048] The proposed satellite tracking device, monitoring system, computer program and information medium have the advantages described above. above in connection with the proposed satellite tracking method. Brief description of the drawings

[0049] Other characteristics and advantages of the present invention will emerge from the description provided below, illustrating embodiments of the invention given by way of example and without any limiting character, with reference to the attached drawings:

[0050] [Fig.l] represents an example of architecture of a monitoring system according to an embodiment of the invention;

[0051] [Fig.2] represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention;

[0052] [Fig.3] represents steps of a method for tracking satellites in a sequence of images according to an embodiment of the invention;

[0053] [Fig.4] represents steps of a method according to an embodiment of the invention for eliminating stars in a sequence of images;

[0054] [Fig.5] represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention;

[0055] [Fig.6] represents examples of images processed by a satellite tracking device according to one embodiment of the invention;

[0056] [Fig.7] represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention; and

[0057] [Fig. 8] represents steps of a method according to an embodiment of the invention for carrying out post-processing operations on satellite trajectories. Description of the embodiments

[0058] The present invention applies, in particular, to space surveillance systems. The following description of the invention will refer to this particular context of application, which is given only as an illustrative example and should not limit the invention.

[0059] [Fig.l] represents an example of architecture of a monitoring system according to an embodiment of the invention. This figure is described below to introduce the present invention and to exemplify an application context.

[0060] The SYS monitoring system comprises at least one SENS acquisition device and one APP satellite tracking device. The SYS system may also comprise a restitution and / or storage device OUT.

[0061] We describe below a system SYS comprising a single device SENS acquisition devices. However, embodiments could also be envisaged in which the SYS system comprises a plurality of SENS acquisition devices, located at different locations on Earth and / or of different types.

[0062] The SENS device is configured to acquire (i.e. capture) a sequence of images IMG_SEQ. This SENS device comprises one or more optical sensors, such as a camera, or an infrared sensor. As illustrated in [Fig.l], the SENS device is configured to provide the sequence of images IMG_SEQ to the APP device.

[0063] The image sequence IMG_SEQ comprises a plurality of consecutive images covering a portion of the sky. Each image of the sequence IMG_SEQ is associated with an acquisition time. Typically, within the images of the sequence IMG_SEQ, there are satellites orbiting the Earth, stars (in large numbers compared to the satellites), and measurements from noise (e.g. hot pixels).

[0064] It should also be noted that the satellites observed in the image sequence IMG_SEQ may include fixed geostationary satellites, drifting geostationary satellites, and moving satellites. In other words, the observed satellites may have geostationary orbits GEO (acronym for “Geosynchronous Earth Orbit” in English), medium Earth orbits MEO (acronym for “Medium Earth Orbit” in English), or low Earth orbits LEO (acronym for “Low Earth Orbit” in English).

[0065] The APP device is configured to track the satellites observed in the image sequence IMG_SEQ. In other words, it is configured to determine the respective TRJ trajectories of the different satellites present in the image sequence IMG_SEQ.

[0066] In the context of the invention, a "satellite trajectory" is defined by a set of measurements associated with the satellite, each of these measurements comprising a position of the satellite (e.g. x and y coordinates of the satellite in the image), and a time index (e.g. index of the image in the sequence, or instant of acquisition of the image). For example, a satellite trajectory Tm can be defined as follows: Tm = {(xn, yn, in) I n G Nm)}, with Nm the indices of the measurements associated with this trajectory.

[0067] According to the embodiment illustrated by [Fig.l], the APP device is configured to provide the TRJ satellite trajectories to the OUT device. For example, the OUT device may be a rendering device comprising a screen for displaying the TRJ satellite trajectories. Alternatively, the OUT device could be a storage device used to record the TRJ satellite trajectories.

[0068] The APP device comprises, according to the embodiment illustrated by [Fig.l]: at least one processing unit or processor PROC; and at least one memory MEM.

[0069] More particularly, the device APP has, according to one embodiment, the hardware architecture of a computer. The memory MEM constitutes an information medium in accordance with the invention, that is to say readable by the processor PROC and on which a computer program PROG in accordance with the invention is recorded. The The PROG program comprises instructions for carrying out steps of a satellite tracking method according to the invention, when the PROG program is executed by the PROC processor. The PROG program defines in particular the functional modules of the APP device shown in [Fig. 2], which rely on or control the hardware elements of the latter.

[0070] As illustrated by [Fig.l], the APP device has a COM communication module configured to communicate with the SENS acquisition device and / or the OUT device. No limitation is attached to the nature of the communication interfaces between these devices, which may be wired or wireless, and may implement any protocol known to those skilled in the art.

[0071] The architecture of the APP satellite tracking device having been introduced, we describe below its operation with reference to the following figures.

[0072] [Fig.2] and [Fig.3] respectively represent an example of architecture of a satellite tracking device and steps of a satellite tracking method according to an embodiment of the invention. These figures present the operation of the APP device introduced with reference to [Fig.l].

[0073] The various functional modules of the APP device are illustrated in [Fig.2]. We describe the functions of these modules in relation to the steps of the method in [Fig.3],

[0074] As illustrated by [Fig.3], the proposed method comprises all or part of the steps S100 to S600 described below, these being implemented by the APP device.

[0075] The proposed method may comprise a plurality of iterations of each of its steps. The steps of the method may be repeated at a given frequency, for example every night, or every week.

[0076] In step S100, the obtaining module OBT of the APP device obtains the image sequence IMG_SEQ. From this, it obtains a set of measurements S1N associated with the spatial objects observed in the images of the sequence IMG_SEQ.

[0077] The APP device receives the image sequence IMG_SEQ from the SENS acquisition device, via its communication module COM. Alternatively, the APP monitoring device could also read the image sequence IMG_SEQ from a memory shared with the SENS acquisition device.

[0078] Then, the APP device processes the image sequence IMG_SEQ to obtain the set of S1N measurements associated with the different spatial objects observed in the sequence IMG_SEQ. To do this, the APP device can use any known technique for detecting points in images. For example, it can use a high-pass filter, or a neural network.

[0079] For each spatial object observed in an image of the IMG_SEQ sequence, the APP device obtains a measurement (i.e. a point) in three dimensions. Each measurement includes x and y coordinates describing the position of the object in the image, and a time index (e.g. the index of the image in the sequence, the time of image acquisition). Thus, the set of measurements Slx={(xri, yn, in) I ne N1n} with N1n the indices of the measurements obtained.

[0080] In step S200, the FLT filter of the APP device filters the set of measurements SIX to eliminate the measurements associated with stars. The APP device thus obtains a filtered set of measurements SFLT={(xn, yn, in) I ne NFLT] with NFLT the indices of the retained measurements. Thus, the filtered set of measurements SFLT does not include the measurements considered as corresponding to stars.

[0081] We describe in more detail the implementation of step S200 with reference to [Fig.4],

[0082] In step S300, the DET detector of the APP device detects, from the filtered measurement set SFLT, the satellite trajectories TRJ in the image sequence IMG_SEQ. The DET detector comprises (i.e. implements) at least one artificial neural network.

[0083] In step S400, the CLA classifier of the APP device classifies the detected satellite trajectories TRJ. The CLA classifier comprises (i.e. implements) at least one artificial neural network.

[0084] Following steps S300 and S400, the APP device obtains satellite trajectories TRJ. As mentioned previously, the trajectory of a satellite Tm is defined by a set of measurements Tm={(xn, yn, in) I ne Nm] with Nm the indices of the trajectory measurements. In addition, the APP device assigns a class Cm to this trajectory Tm.

[0085] The classes assigned to the trajectories by the APP device indicate the types of trajectories (i.e. orbits) of the satellites. For example, the list of assignable classes may include: fixed geostationary trajectory; drifting geostationary trajectory; scrolling type trajectory (i.e. satellite in medium or low Earth orbit moving at high speed); and trailing type trajectory (i.e. satellite in low Earth orbit moving at very high speed); and noise.

[0086] The implementation of steps S300 and S400 is described in more detail with reference to Figures 5 and 7. More specifically, [Fig.5] illustrates a first embodiment of steps S300 and S400, while [Fig.7] illustrates a second embodiment of these steps.

[0087] In step S500, the processing module PP of the device APP performs (analytical) operations known as post-processing of the detected satellite trajectories TRJ.

[0088] This step makes it possible in particular to improve the results of the proposed solution for tracking satellites in a sequence of images IMG_SEQ. The implementation of step S500 is detailed below with reference to [Fig.8].

[0089] In step S600. and according to one embodiment, the APP device provides the tra TRJ satellite receivers to the OUT device, via its COM communication module.

[0090] As indicated previously, each of the satellite trajectories TRJ provided by the APP device comprises: a set of measurements Tm describing the positions of a satellite in the images of the sequence IMG_SEQ; and a class Cm indicating the type of trajectory of the satellite.

[0091] The OUT device can for example display the trajectories of TRJ satellites on a screen. Alternatively, this OUT device can be used to store the trajectories of TRJ satellites. These TRJ trajectories can be analyzed later by a separate device in order to catalog the orbits of the observed satellites.

[0092] We have presented here the general operation of the APP device proposed for tracking satellites in an image sequence IMG_SEQ. Also, we detail below the implementation of steps S200 to S500 with reference to the following figures.

[0093] [Fig.4] represents steps of a method according to an embodiment of the invention for eliminating stars in a sequence of images. This figure details the implementation of the filtering step S200 for eliminating the measurements associated with stars.

[0094] As illustrated by [Fig.4], step S200 comprises all or part of steps S210 to S240 described below and implemented by the FLT filter of the APP device.

[0095] In step S210, the APP device selects, from the set of measurements Sin, the measurements whose brightness is greater than a threshold in the images of the sequence IMG_SEQ. Thus, the APP device selects the brightest points in the images of the sequence IMG_SEQ.

[0096] The inventors found that the majority of the brightest measurements in an image sequence are stars. For this reason, the APP device uses the brightest measurements to determine star motion information (either a star motion speed per image or a star rotation angle per image).

[0097] In step S220, the APP device determines the star displacement information from selected measurements.

[0098] First, the APP device determines whether the trajectories of the stars are substantially rectilinear in the image sequence IMG_SEQ, or whether they are substantially circular. To do this, it can determine whether the normals of the trajectories of the stars intersect near an observation center (i.e., whether the intersection of these normals is close to the observation center).

[0099] In fact, if the image sequence IMG_SEQ is acquired by an observatory located near the equator, the trajectories of the stars are substantially rectilinear. The APP device then determines the average speed of the stars per image. In other words, it determines the displacement vector of the stars from one image to the next (e.g. -5.1 pixels / image along the axis, and -1.2 pixels / image along the y axis). To determine this displacement vector, the APP device calculates the average of the distances between each of the measurements and its nearest neighbor in the next image.

[0100] On the other hand, if the image sequence IMG_SEQ is acquired by an observatory located near the poles, the trajectories of the stars are substantially circular. In this case, the APP device determines the center of rotation of the stars (i.e. the intersection of the normals), as well as the rotation angle of the stars from one image to another (e.g. 3.2 degrees / image).

[0101] In step S230, the APP device eliminates, using the star displacement information, the measurements considered to come from stars.

[0102] According to one embodiment, the APP device performs step S230 in the following manner. For each observed spatial object, the APP device predicts its positions in the other images of the sequence by assuming that this object is a star. If these predictions prove to be correct, then the APP device considers that the object is a star and eliminates the associated measurement.

[0103] More precisely, for each measurement of each image of the sequence, the APP device predicts the positions of the object in the other images of the sequence using the star displacement information. If the number of predicted positions considered correct is greater than a threshold, the APP device eliminates the measurement considered. A predicted position is considered correct when there is a measurement present in the image and whose distance to the predicted position is less than a threshold.

[0104] In step S240, the APP device eliminates, among the remaining measurements, the isolated measurements in the image sequence IMG_SEQ.

[0105] Specifically, the APP device identifies measurements that do not have close neighbors, and eliminates these measurements. A measurement is considered to have no close neighbors when there are no measurements located near it, i.e., no measurements below a distance threshold. The APP device considers in particular the image in which the measurement is located, but also the previous and following images (e.g., the two previous images, and the two following images).

[0106] According to one embodiment, the APP device can also eliminate the measurements located on the edges (i.e. included below a given distance from the edges of the images) in the first images or the last images of the IMG_SEQ sequence.

[0107] Following these steps, the APP device obtains a set of filtered SFLF measurements, in which the measurements considered to be associated with stars have been eliminated.

[0108] This SFLF set is then used by the APP device to detect tra- TRJ satellite ejectories in the image sequence IMG_SEQ. In particular, we describe with reference to Figures 5 and 7 two embodiments of the detection and classification of satellite trajectories.

[0109] [Fig.5] represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention. This figure illustrates a first embodiment of the DET detector and the CLA classifier, previously introduced with reference to Figures 2 and 3.

[0110] As mentioned previously, during step S300, the detector DET detects satellite trajectories TRJ in the image sequence IMG_SEQ from the filtered measurement set SFLT. Then, during step S400, the classifier CLA classifies (i.e. segments) the trajectories TRJ.

[0111] This first embodiment proposes to use convolutional neural networks to implement the DET detector, and to use PointNet type neural networks to implement the CLA classifier.

[0112] The DET detector comprises, according to this first embodiment, several convolutional neural networks CNNLs, CNNhs, and CNNvhs - Each of these networks takes as input an image in which all the SFLF filtered measurements are superimposed, and provides as output BB frames encompassing detected satellite trajectories (i.e. "bounding boxes" in English). Such a frame Bm can for example be defined in the following manner: Bm={xm, ym, hm, lm] with xm and ym the coordinates of the center of the frame, hm its height and lm its width.

[0113] The CNNLS, CNNHs, and CNNVhs neural networks are respectively dedicated (i.e. specialized, trained) to the detection of satellite trajectories of different speeds. In particular, the CNNLS network is trained to detect satellite trajectories in geostationary orbit that are substantially fixed or drifting (i.e. low-speed trajectories). The CNNHs network is trained to detect satellite trajectories in medium or low orbit and moving at high speed (i.e. scrolling type trajectories). Finally, the CNNvhs network is trained to detect satellite trajectories in low orbit and moving at very high speed (i.e. trailing type trajectories).

[0114] The CNNLS network can use a Faster R-CNN type neural network with an Inception V2 type feature extraction network (i.e. "backbone network"). Faster R-CNN type networks and their implementation are for example described in the following document: Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", 2016. And, Inception V2 type networks and their implementation are for example described in the following document: Szegedy et al., "Rethinking the Inception Architecture for Computer Vision", 2015.

[0115] In addition, the CNNHs and CNNVhs networks can respectively use a Faster R-CNN type neural network with a Resnet 101 type feature extraction network. The ResNet 101 type networks and their implementation are for example described in the following document: He et al., “Deep Residual Learning for Image Recognition”, 2015.

[0116] Training of the DET detector neural networks is performed using simulated training data, i.e., images comprising computer-generated satellite trajectories to which noise is added. The neural networks can also be pre-trained using public datasets, such as the "COCO" dataset.

[0117] More particularly, to train the neural networks of the proposed solution, it may be envisaged to generate data comprising trajectories of stars (circular or rectilinear), trajectories of space objects of different types, and noise. Then, the measurements are filtered to eliminate the stars using the technique previously described with reference to [Fig.3]. Finally, the measurements thus obtained are used to train the neural networks.

[0118] To dedicate the CNNls network to the detection of satellite trajectories in substantially fixed or drifting geostationary orbit (i.e. low-speed trajectories), it is trained using only these types of trajectories. Similarly, the CNNhs network is trained using scrolling type trajectories, and the CNNvhs network is trained using trailing type trajectories.

[0119] For implementation details concerning the training of neural networks, in particular the loss functions, those skilled in the art may refer to the aforementioned documents and the references cited in these documents.

[0120] According to a particular embodiment, the CNNLS neural network processes in parts the image in which the set of SFLT filtered measurements is superimposed. On the other hand, the CNNHs and CNNvhs networks take the entirety of this image as input.

[0121] The CLA classifier comprises, according to this first embodiment, several PNLS and PNHs neural networks of PointNet type. These PNLS and PNHs networks are respectively dedicated to the classification (i.e. the segmentation) of trajectories of TRJ satellites of different speeds.

[0122] On the one hand, the PNLS network is trained to classify trajectories of satellites in geostationary orbit as substantially fixed or drifting (i.e. low-speed trajectories). As illustrated in [Fig.7], the PNLs network takes as input the BB frames provided by the CNNls network, dedicated to the detection of low-speed trajectories.

[0123] The NLP network processes each of the BB frames separately. For each frame Bm encompassing a trajectory Tm, it takes as input all the measurements located inside the frame Bm in the image sequence IMG_SEQ. Note also that the measurements located in the Bm frame are recentered before providing them to the PNLS network so that the xm and ym coordinates of the center of the frame are equal to 0.

[0124] The PNLS network provides as output a label for each of the measurements located in the Bm frame, this label indicating whether or not the measurement belongs to the Tm trajectory. Thus, the PNLS network makes it possible to separate the measurements belonging to the TRJ trajectories from the noise. In addition, it provides as output a class Cm attributed to the Tm trajectory.

[0125] The list of classes attributable by the PNLS network is: fixed geostationary trajectory; drifting geostationary trajectory; double drifting geostationary trajectories (two drifting trajectories are present in the same frame); and noise.

[0126] On the other hand, the PNHs network is trained to classify trajectories of satellites in medium or low orbit (i.e. high-speed trajectories). As illustrated in [Fig.l], the PNHs network takes as input the BB frames provided by the CNNHs and CNNvhs networks, dedicated to the detection of high-speed trajectories.

[0127] The PNHs network processes each of the BB frames separately. For the PNHs network, only the measurements around the diagonals of the BB bounding frames (i.e., those included below a distance threshold from the diagonals) are provided. These measurements are recentered before providing them to the PNHs network so that the xm and ym coordinates of the center of the frame are equal to 0. In addition, these measurements are normalized before providing them to the PNHs network, by dividing each of the coordinates of the measurements by the maximum absolute value of the x and y coordinates.

[0128] The PNHs network provides as output a label for each of the measurements located in the frame Bm, this label indicating whether or not the measurement belongs to the trajectory Tm. It also provides as output a class Cm attributed to the trajectory Tm.

[0129] The list of classes assignable by the PNHs network is: high-speed trajectory (scrolling-type trajectory or trailing-type trajectory); and noise. Then, the APP device can distinguish between scrolling-type trajectories and trailing-type trajectories as follows. Scrolling-type trajectories comprise on average less than 1.5 measurements per image, while trailing-type trajectories comprise on average more than 1.5 measurements per image.

[0130] For each bounding box Bm, the APP device thus obtains: a set of measurements associated with the trajectory Tm={(xn, yn, in) I ne Nm] with Nm the indices of the measurements of the trajectory; and a class Cm attributed to the trajectory Tm.

[0131] PointNet type networks and their implementation are for example described in the previously cited Qi et al. document. PointNet type NLPs and PNHs networks are trained using simulated data.

[0132] To dedicate the PNLs network to the classification (segmentation) of satellite trajectories in substantially fixed or drifting geostationary orbit (i.e. low-speed trajectories), it is trained using only these types of trajectories. Similarly, the PNHs network is trained using scrolling type trajectories and trailing type trajectories (i.e. high-speed trajectories).

[0133] For illustration purposes, the following figure represents the data processed and obtained by the different modules of the APP device according to this first embodiment.

[0134] [Fig.6] represents examples of images processed by a satellite tracking device according to one embodiment of the invention.

[0135] The first image IMG_SUP represents a superposition of the images of the sequence IMG_SEQ obtained by the obtaining module OBT during step S100.

[0136] The second image IMG_SUP_FLT represents within the same image the set of filtered SFLT measurements obtained by the FLT filter following step S200.

[0137] The third image IMG_TRJ_DET represents within the same image the BB frames encompassing the satellite trajectories detected by the DET detector following step S300.

[0138] The fourth image IMG_TRJ_CLA represents within the same image the trajectories of TRJ satellites obtained by the CLA classifier following step S400.

[0139] The fifth image IMG_TRJ_PP represents within the same image the trajectories of satellites TRJ obtained by the processing module PP following step S500.

[0140] We have described above a first embodiment of the DET detector and the CLA classifier of the APP device. However, within the framework of the invention, other embodiments can be envisaged. Also, we describe below with reference to [Fig.7] a second embodiment of the detection and classification of satellite trajectories.

[0141] [Fig.7] represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention. This figure illustrates a second embodiment of the DET detector and the CLA classifier, previously introduced with reference to Figures 2 and 3.

[0142] The DET detector comprises, according to this second embodiment, at least one Transformer-type TRF neural network, i.e. a network comprising an attention mechanism. Transformer-type TRF neural networks and their implementation are for example described in the document A. Vaswani et al. cited above.

[0143] According to this second embodiment, the TRF network of the DET detector takes as input the measurements of the SFLF set, and provides a latent vector LV for each of these measurements. Then, a CLST association module of the DET detector associates (i.e. groups together) the measurements belonging to the same satellite trajectory Tm according to the Euclidean distances between the latent vectors LV.

[0144] The APP device thus detects the respective TRJ trajectories of the different sa- tellites observed in the image sequence IMG_SEQ. More precisely, for each detected trajectory Tm, the APP device obtains a set of measurements associated with the trajectory Tm={(xn, yn, in) I ne Nm} with Nm the indices of the trajectory measurements.

[0145] As an example, the CLST association module uses a technique called agglomerative hierarchical clustering with a defined distance threshold. The details of implementing the agglomerative hierarchical clustering technique are known to those skilled in the art.

[0146] However, within the framework of the invention, embodiments could be envisaged in which the CLST association module uses other grouping algorithms to associate the measurements belonging to the same satellite trajectory from the distances between the latent vectors.

[0147] The CLA classifier comprises, according to this second embodiment, at least one classification neural network CN. This CN neural network separately processes the different detected satellite trajectories TRJ. For each trajectory Tm, the CN classification network takes as input the measurements associated with this trajectory Tm, and provides as output a class Cm attributed to this trajectory Tm.

[0148] No limitation is attached to the architecture of the CN classification neural network. For example, the CN network may be a multi-layer perceptron, comprising several dense layers with ReLU-type activation functions, and an output layer with a Softmax-type activation function.

[0149] It should be noted that TRF and CN neural networks can be trained from simulated (i.e. computer-generated) training data.

[0150] In the context of the invention, embodiments could also be envisaged in which the DET detector comprises a plurality of Transformer-type neural networks respectively dedicated to the detection of trajectories of TRJ satellites of different speeds. Similarly, the CLA classifier could comprise a plurality of classification neural networks respectively dedicated to the classification of trajectories of TRJ satellites of different speeds.

[0151] Following step S300 of detection and step S400 of classification of the satellite trajectories TRJ, the APP device performs during step S500 various post-processing operations of these trajectories TRJ. As mentioned previously, step S500 makes it possible to improve the reliability of the satellite tracking carried out by the APP device. We detail the implementation of this step with reference to the following figure.

[0152] [Fig.8] represents steps of a method according to an embodiment of the invention for carrying out post-processing operations on satellite trajectories. This figure details the implementation of step S500 introduced with reference to Figures 2 and 3.

[0153] As illustrated by [Fig.8], and according to one embodiment, step S500 comprises all or part of steps S510 to S550 described below and implemented by the processing module PP of the APP device.

[0154] In step S510, the APP device merges (i.e. combines) satellite trajectories TRJ to group the measurements associated with several detected satellite trajectories TRJ and belonging to the same satellite. Thus, if measurements belonging to the same satellite have been associated with several trajectories by the APP device, then these measurements are grouped within the same trajectory.

[0155] To do this, the APP device compares the trajectories of TRJ satellites two by two, and merges them or not depending on the result of the comparison.

[0156] Specifically, the APP device compares the number of measurements common to two trajectories Tm and Tp. If this number is greater than a given percentage (eg 20%) of the number of measurements of the smallest of the two trajectories, then the measurements of the two trajectories Tm and Tp are grouped within a single trajectory. In other words, the APP device then performs the union of the two trajectories Tm and Tp.

[0157] It is important to note that the APP device does not compare all types of trajectories together. On the one hand, fixed geostationary type trajectories are compared only with each other. On the other hand, drifting geostationary trajectories, scrolling type trajectories, and trailed type trajectories are compared together.

[0158] In step S520, the APP device completes the trajectories of TRJ satellites of scrolling types, i.e. high-speed trajectories comprising on average less than 1.5 measurements per image. This completion uses three-dimensional linear regressions.

[0159] For a trajectory Tm of a satellite, the completion carried out by the APP device makes it possible to eliminate measurements associated with the trajectory Tm, but not belonging to this satellite. In addition, this completion makes it possible to add measurements belonging to the satellite and which would not have been associated with the trajectory Tm.

[0160] For each measurement of the trajectory Tm, the APP device predicts the position of the object using three-dimensional linear regression determined from the measurements of the trajectory Tm. If there is another measurement closer to the predicted position than the measurement considered, then the measurement considered is eliminated from the trajectory Tm, and is replaced by the other closer measurement.

[0161] For each image ik of the sequence IMG_SEQ, the APP device proceeds as follows. If the trajectory Tm includes a measurement on this image ik, and if the distance between this measurement and a predicted position is greater than a threshold, then the measurement is eliminated from the trajectory Tm. If the trajectory Tm does not include a measurement on this image ik, and if the distance between the predicted position and the measurement closest to the predicted position is less than a threshold, then this measurement is associated (i.e. added) to the trajectory Tm.

[0162] In step S530, the APP device filters the trail-type TRJ trajectories, i.e. the high-speed trajectories comprising on average more than 1.5 measurements per image.

[0163] For each drag-type trajectory Tm, the APP device determines the slope and the ordinate at the origin of the linear regression line formed by the measurements of the trajectory Tm. It then eliminates, among the measurements of the trajectory Tm, those which deviate beyond a given distance from the linear regression line.

[0164] The APP device thus eliminates measurements that are not aligned along the two-dimensional trajectory. In this sense, it performs a polishing of the trail-type trajectories.

[0165] In step S540, the APP device merges the trail-type trajectories. This makes it possible to prevent the same trail-type trajectory from being detected in several disjoint pieces by the APP device. For this purpose, the APP device compares the trail-type trajectories with each other, and merges them or not depending on the result of the comparison.

[0166] In fact, the device compares two by two the linear regression lines of the trail type trajectories. For two trail type trajectories Tm and Tp, if the difference between their slopes is less than a threshold, and if the difference between their ordinates at the origin is less than another threshold, then the APP device groups the measurements of the two trajectories Tm and Tp within a single trajectory.

[0167] In step S550, the APP device filters the satellite trajectories TRJ so that each of the trajectories comprises only one measurement per image of the sequence IMG_SEQ.

[0168] If a trajectory Tm comprises several measurements on an image of the sequence IMG_SEQ, the APP device combines these measurements to obtain a single measurement. For example, the position of the single measurement obtained by the APP device may correspond to the barycenter of the positions of the measurements of the trajectory Tm included in this image.

[0169] Additional variants: It should be noted that the order in which the steps of a method according to the invention are carried out, in particular with reference to the attached drawings, constitutes only an example of embodiment without any limiting character, variants being possible. In particular, a method according to the invention may comprise one or more iterations of the steps described above, in particular with reference to the attached drawings. Furthermore, the reference signs do not limit the scope of the protection, their sole function being to facilitate the understanding of the claims.

[0170] A person skilled in the art will understand that the embodiments and variants described above constitute only non-limiting examples of implementation of the invention. In particular, those skilled in the art may envisage any adaptation or combination of the embodiments and variants described above in order to meet a very specific need.

Claims

Claims

1. Method for tracking satellites in a sequence of images (IMG_SEQ), the method comprising: - a step of obtaining (S 100) a set of measurements (S^) associated with spatial objects observed in the images of the sequence (IMG_SEQ), each measurement comprising a position of an object in an image and a temporal index of the image; - a step of detecting (S300) satellite trajectories (TRJ) in the sequence of images (IMG_SEQ), by a detector (DET) comprising at least one neural network and using all or part of the set of measurements obtained (S1N); and - a step of classifying (S400) the detected satellite trajectories (TRJ), by a classifier (CLA) comprising at least one other neural network.

2. Method according to claim 1, in which: - the detector (DET) comprises at least one convolutional neural network (CNN), and determines (S200) frames (BB) each encompassing a satellite trajectory (TRJ) in the images of the sequence (IMG_SEQ); and - the classifier (CLA) comprises at least one PointNet type neural network, and for each frame (BB) encompassing a satellite trajectory (TRJ), the classifier (CLA) determines (S300) the measurements located in the frame belonging to this trajectory, and assigns (S300) a class to this trajectory.

3. Method according to claim 1, in which the detector (DET) comprises at least one Transformer type neural network (TRF), and associates (S200) the measurements belonging to the same satellite trajectory (TRJ) in the images of the sequence (IMG_SEQ).

4. Method according to claim 3, in which the detector (DET) determines (S200) latent vectors (LV) representative of said measurements (SFLT) using said at least one Transformer-type neural network, and associates (S200) measurements with the same trajectory of satellite (TRJ) as a function of distances between latent vectors (LV).

5. Method according to any one of claims 1 to 4, wherein: - the detector (DET) comprises a plurality of neural networks (CNNLS, CNNHs, CNNVhs) respectively dedicated to the detection of satellite trajectories (TRJ) of different speeds; and - the classifier (CLA) comprises a plurality of neural networks (PNLS, PNHs) respectively dedicated to the classification of satellite trajectories (TRJ) of different speeds.

6. Method according to any one of claims 1 to 5, comprising a step of filtering (S200) the set of measurements (S^) to eliminate the measurements associated with stars, and in which the detector (DET) uses only the filtered set of measurements (SFLT) to carry out the step of detecting (S300) satellite trajectories (TRJ) in the sequence of images (IMG_SEQ).

7. Method according to claim 6, in which the step of filtering (S200) the set of measurements (SiN) to eliminate the measurements associated with stars comprises: - a determination (S220) of displacement information of the stars from measurements whose brightness is greater than a threshold in the images of the sequence (IMG_SEQ); and - a step of eliminating (S230) the measurements considered to be associated with stars from the determined displacement information.

8. Method according to any one of claims 1 to 7, further comprising: - one or more steps of merging (S510, S540) the satellite trajectories to group the measurements associated with several detected satellite trajectories (TRJ) and belonging to the same satellite; and - a step of filtering (S550) the satellite trajectories (TRJ) so that a said satellite trajectory (TRJ) comprises only one measurement per image of the sequence (IMG_SEQ).

9. Device (APP) for tracking satellites in a sequence of images (IMG_SEQ), the device (APP) comprising: - an obtaining module (OBT) configured to obtain (S 100) a set of measurements (SiN) associated with spatial objects observed in the images of the sequence (IMG_SEQ), each measurement comprising a position of an object in an image and a temporal index of the image; - a detector (DET) configured to detect (S300) satellite trajectories (TRJ) in the sequence of images (IMG_SEQ), the detector (DET) comprising at least one neural network and using all or part of the set of measurements obtained (Sjn); and - a classifier (CLA) configured to classify (S400) the detected satellite trajectories (TRJ), the classifier (CLA) comprising at least one other neural network.

10. Surveillance system (SYS) comprising: - a device (APP) for tracking satellites according to claim 9; and - a device (SENS) for acquiring a sequence of images (IMG_SEQ).

11. Computer program (PROG) comprising instructions for implementing the steps (S100-S600) of a method according to one of claims 1 to 8, when said computer program (PROG) is executed by at least one processor (PROC).

12. Computer-readable information medium (MEM) comprising a computer program (PROG) according to claim 11.

Images