Method and device for tracking satellites in an image sequence

Abstract

The present invention relates to a method and a device (APP) for tracking satellites in an image sequence (IMG_SEQ). In particular, the method comprises: - a step of obtaining a set of measurements (SIN) associated with space objects observed in the images of the sequence (IMG_SEQ), each measurement comprising a position of an object in an image and a time index of the image; - a step of detecting satellite trajectories (TRJ) in the image sequence (IMG_SEQ) by means of a detector (DET) comprising at least one neural network and using all or part of the obtained set of measurements (SIN); and - a step of classifying the detected satellite trajectories (TRJ) by means of a classifier (CLA) comprising at least one other neural network.

Description

Description Title: Method and device for tracking satellites in an image sequence Technical field

[0001] The present invention relates to the fields of image analysis and object tracking. In particular, the present invention relates to a method and device for tracking satellites in an image sequence, as well as an associated monitoring system, computer program and information medium. The present invention finds a particularly advantageous, although in no way limiting, application for the implementation of space surveillance systems. Prior art

[0002] The invention falls within the specific context of outer space surveillance systems. Such systems aim to locate satellites in orbit and provide their respective trajectories. To this end, these systems use optical sensors to capture images of satellites orbiting the Earth. However, these images include satellites, stars, and noise (e.g., hot pixels).

[0003] Therefore, we are interested below in surveillance systems capable of extracting the trajectories of the different satellites observed in a sequence of images. In other words, these systems aim to provide as output a set of measurements associated with the same satellite to describe its trajectory over time, that is to say a set of three-dimensional coordinates associated with this satellite (spatial positions x and y, and temporal index).

[0004] In the current state of the art, there are analytical solutions for tracking satellites within an image sequence. These analytical solutions exploit the distances between measurements associated with observed space objects to extract satellite trajectories and detect different types of trajectories (e.g. geostationary, moving).

[0005] However, existing satellite tracking solutions have a number of drawbacks, including the following. On the one hand, these analytical solutions involve a high implementation complexity, which requires the manual configuration of a multitude of parameters and a significant use of computing resources (i.e., significant execution times). On the other hand, the reliability of these solutions, in terms of correctly detected trajectories, is not fully satisfactory.

[0006] There is therefore a need for a solution that can track satellites in an image sequence reliably and with minimal implementation complexity. Statement of the invention

[0007] The present invention aims to remedy all or part of the drawbacks of the prior art, in particular those set out above.

[0008] According to one aspect of the invention, there is provided a method for tracking satellites in a sequence of images, the method comprising: a step of obtaining a set of measurements associated with spatial objects observed in the images of the sequence, each measurement comprising a position of an object in an image and a temporal index of the image; a step of detecting satellite trajectories in the sequence of images, by a detector comprising at least one neural network and using all or part of the set of measurements obtained; and a step of classifying the detected satellite trajectories, by a classifier comprising at least one other neural network.

[0009] The present invention makes it possible to extract the respective trajectories of the satellites observed in the image sequence. In fact, the present invention makes it possible to associate the measurements belonging to the same satellite to describe its trajectory in the image sequence. It is important to note that the present invention makes it possible not only to track satellites in geostationary orbit, but also satellites in medium or low Earth orbit.

[0010] In addition, the present invention allows for classifying the detected trajectories. In particular, the class assigned to each satellite trajectory may indicate the type of orbit of the satellite. For example, the list of assignable classes may include: fixed geostationary trajectory; drifting geostationary trajectory; scrolling type trajectory (i.e. satellite in medium or low Earth orbit moving at high speed); and trailing type trajectory (i.e. satellite in low Earth orbit moving at very high speed).

[0011] Compared to existing analytical solutions, the present invention makes it possible to improve the reliability of tracking observed satellites within an image sequence. It also has reduced implementation complexity (i.e. reduced execution times, less configuration). Advantageously, the present invention does not require a priori knowledge of the orbits to track satellites in an image sequence.

[0012] Indeed, the use of one or more neural networks to implement the detector makes it possible to obtain reliable detection of trajectories, in terms of correctly detected trajectories. The neural network(s) are trained automatically from training data (i.e. reference image sequences) to optimize the reliability of satellite trajectories detection. This makes it possible in particular to obtain significantly improved performance compared to analytical solutions. In addition, the use of one or more neural networks multiple neural networks can significantly reduce the computing resources required, particularly in terms of execution time.

[0013] The same applies to the classifier comprising one or more neural networks, which improves the reliability of trajectories classification (i.e. correctly identified trajectories). This also reduces the necessary computing resources (i.e. execution time). The joint use of one or more neural networks to detect satellite trajectories, and one or more other neural networks to classify these trajectories makes it possible to obtain reliable tracking of the satellites observed in the image sequence.

[0014] Thus, the present invention makes it possible to track satellites in an image sequence reliably and with minimal implementation complexity.

[0015] According to one embodiment, the detector comprises at least one convolutional neural network, and determines frames each encompassing (surrounding) a satellite trajectory in the images of the sequence. In addition, according to this embodiment, the classifier comprises at least one classification (i.e. segmentation) neural network, in particular at least one PointNet type neural network. And, for each frame encompassing a satellite trajectory, the classifier determines (i.e. associates) the measurements located in the frame belonging to this trajectory, and assigns a class to this trajectory.

[0016] By "convolutional neural network" we mean a neural network comprising one or more convolutional layers. And by "PointNet neural network" we mean a neural network dedicated to processing point clouds, and in particular to classification and segmentation applications. PointNet networks and their implementation are described, for example, in the following document: Qi et al., "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation", 2017.

[0017] This implementation mode proposes to use one or more convolutional neural networks to detect satellite trajectories, and to use one or more PointNet type neural networks to classify (segment) these trajectories.

[0018] The convolutional neural network(s) allow the detector to exploit the spatio-temporal characteristics of the image sequence to detect satellite trajectories. More precisely, it exploits the spatio-temporal characteristics of the image sequence to determine bounding boxes for the satellite trajectories. The convolutional neural network(s) thus make it possible to obtain reliable detection of the trajectories of the different satellites observed in the image sequence.

[0019] In addition, using one or more PointNet neural networks to implement the classifier is particularly advantageous. Indeed, the PointNet network is optimized for classifying and segmenting point clouds. Here, the PointNet network processes each of the frames encompassing the satellite trajectories separately. For each frame encompassing a trajectory, it takes as input all the measurements located inside the bounding box in the image sequence. Thus, these PointNet type networks make it possible to reliably separate the measurements belonging to the trajectory from those not belonging to the trajectory, and to assign a class to this trajectory.

[0020] Also, this mode of implementation makes it possible to reliably extract the respective trajectories of the different satellites observed in the sequence of images and to classify them.

[0021] According to one embodiment, the detector comprises at least one Transformer-type neural network, and associates the measurements belonging to the same satellite trajectory in the images of the sequence. Furthermore, according to this embodiment, the classifier comprises at least one classification network and assigns a class to each of the satellite trajectories.

[0022] Here, we refer to a "Transformer-type neural network" as a neural network that uses an attention mechanism. Transformer-type neural networks and their implementation are described, for example, in the following document: A. Vaswani et al., "Attention Is All You Need", 2017.

[0023] It should be emphasized that Transformer-type networks are optimized to process sequential data and allow the exploitation of the temporal characteristics of the input data. In the context of the invention, the use of a Transformer-type network allows the detector to exploit the temporal characteristics of the set of measurements to extract the respective trajectories of the satellites observed in the image sequence.

[0024] This implementation mode is particularly advantageous in that it improves the reliability of tracking satellites observed in an image sequence. In addition, the use of a Transformer-type network reduces the computing resources (i.e. execution time) required to implement the proposed solution.

[0025] According to one embodiment, the detector determines latent vectors representative of said measurements using said at least one Transformer-type neural network; and the detector associates measurements with the same satellite trajectory as a function of distances between the latent vectors.

[0026] This implementation mode proposes to use the Transformer network to represent in a latent space the measurements associated with the spatial objects observed in the image sequence. For each of the measurements provided as input, the Transformer neural network determines a latent vector.

[0027] Note that the Transformer-type neural network allows learning, based on training data, a latent space in which the measurements belonging to the same trajectory are close. This network thus allows representing the measurements in a latent space so as to minimize the distance between the latent vectors of measurements belonging to the same trajectory. trajectory, and to maximize the distances between the latent vectors of measurements belonging to different trajectories.

[0028] This mode of implementation therefore makes it possible, by using the distances between the latent vectors, to reliably associate (i.e. to group) the measurements belonging to the same trajectory.

[0029] According to one embodiment, the detector comprises a plurality of neural networks respectively dedicated to the detection of satellite trajectories of different speeds; and / or the classifier comprises a plurality of neural networks respectively dedicated to the classification of satellite trajectories of different speeds.

[0030] This implementation mode allows adapting the detection and / or classification of trajectories according to different types of trajectories (e.g. fixed or drifting geostationary orbits, medium and low Earth orbits). This advantageously improves the reliability of the proposed solution.

[0031] According to one embodiment, the method comprises a step of filtering the set of measurements to eliminate the measurements associated with (i.e. originating from) the stars, and in which the detector uses only the filtered set of measurements to carry out the step of detecting satellite trajectories in the sequence of images.

[0032] This implementation method makes it possible to eliminate the stars present in the image sequence. Typically, stars are in large numbers compared to satellites in an image sequence, which can interfere with the detection of satellite trajectories. Also, this implementation method contributes to improving the reliability of the proposed solution for tracking satellites in an image sequence. In addition, and advantageously, the proposed solution for eliminating measurements corresponding to stars does not require a priori knowledge of their positions.

[0033] According to one embodiment, the step of filtering the set of measurements to eliminate the measurements associated with stars comprises: a determination of star displacement information from measurements whose brightness is greater than a threshold in the images of the sequence; and a step of eliminating (i.e. deleting) the measurements considered to be associated with stars from the determined displacement information.

[0034] This implementation mode is advantageous in that it allows stars to be reliably eliminated in the sequence images.

[0035] In fact, the inventors observed that the majority of the brightest measurements in an image sequence are stars. Therefore, it is proposed here to exploit the brightest measurements to determine the stars' displacement information (e.g. speed of movement of stars per image, or rotation angle of stars per image). Then, these displacement information is used to determine which measurements correspond to stars, and eliminate these.

[0036] According to one embodiment, the method comprises: one or more steps of merging (i.e. combining) the satellite trajectories to group together the measurements associated with several detected satellite trajectories and belonging to the same satellite; and / or a step of filtering the satellite trajectories so that a said satellite trajectory comprises only one measurement per image of the sequence.

[0037] This implementation mode proposes to carry out different post-processing steps on the satellite trajectories detected by the proposed solution. It thus contributes to improving the reliability of the proposed satellite tracking solution.

[0038] On the one hand, it is proposed to merge (i.e. combine) the trajectories belonging to the same satellite. In other words, if measurements belonging to the same satellite have been associated with a plurality of trajectories, then these measurements are grouped within the same trajectory.

[0039] Furthermore, it is also proposed to filter the different detected satellite trajectories so that each trajectory only includes one measurement per image of the sequence.

[0040] According to another aspect of the invention, there is provided a device for tracking satellites in a sequence of images, the device comprising: an obtaining module configured to obtain a set of measurements associated with spatial objects observed in the images of the sequence, each measurement comprising a position of an object in an image and a temporal index of the image; a detector configured to detect satellite trajectories in the sequence of images, the detector comprising at least one neural network and using all or part of the set of measurements obtained; and a classifier configured to classify the detected satellite trajectories, the classifier comprising at least one other neural network.

[0041] The satellite tracking device may be configured to carry out any of the embodiments of a method according to the invention. In particular, for each step of a method according to the invention, the device may comprise a module configured to implement this step.

[0042] The term module can correspond to a software component as well as to a hardware component or a set of hardware and software components. A software component corresponds to one or more computer programs or subroutines, or more generally to any element of a program capable of implementing a function or a set of functions as described for the modules concerned. In the same way, a hardware component corresponds to any element of a hardware assembly capable of implementing a function or a set of functions for the module concerned.

[0043] According to one aspect of the invention, there is provided a surveillance system comprising: a satellite tracking device according to the invention; and a device for acquiring a sequence of images.

[0044] According to one aspect of the invention, there is provided a computer program comprising instructions for implementing the steps of a method according to the invention, when the computer program is executed by at least one processor or computer.

[0045] The computer program may consist of one or more subparts stored in the same memory or in separate memories. The program may use any programming language, and may be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0046] According to one aspect of the invention, there is provided a computer-readable information medium comprising a computer program according to the invention.

[0047] The information carrier may be any entity or device capable of storing the program. For example, the carrier may comprise a storage means, such as a non-volatile memory or ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard disk. Furthermore, the storage medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by a telecommunications network or by a computer network or by other means. The program according to the invention may in particular be downloaded onto a computer network. Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.

[0048] The proposed satellite tracking device, monitoring system, computer program and information carrier have the advantages described above in connection with the proposed satellite tracking method. Brief description of the drawings

[0049] Other characteristics and advantages of the present invention will emerge from the description provided below, illustrating embodiments of the invention given by way of example and without any limiting character, with reference to the attached drawings:

[0050] Figure 1 represents an example of architecture of a monitoring system according to an embodiment of the invention;

[0051] Figure 2 represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention;

[0052] Figure 3 represents steps of a method for tracking satellites in a sequence of images according to one embodiment of the invention;

[0053] Figure 4 represents steps of a method according to an embodiment of the invention for eliminating stars in a sequence of images;

[0054] Figure 5 represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention;

[0055] Figure 6 represents examples of images processed by a satellite tracking device according to one embodiment of the invention;

[0056] Figure 7 represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention; and

[0057] Figure 8 represents steps of a method according to an embodiment of the invention for carrying out post-processing operations of satellite trajectories. Description of the embodiments

[0058] The present invention applies, in particular, to space surveillance systems. The following description of the invention will refer to this particular context of application, which is given only as an illustrative example and should not limit the invention.

[0059] Figure 1 represents an example of architecture of a monitoring system according to an embodiment of the invention. This figure is described below to introduce the present invention and to exemplify an application context.

[0060] The SYS monitoring system comprises at least one SENS acquisition device and one APP satellite tracking device. The SYS system may also comprise an OUT restitution and / or storage device.

[0061] We describe below a SYS system comprising a single SENS acquisition device. However, embodiments could also be envisaged in which the SYS system comprises a plurality of SENS acquisition devices, located at different locations on Earth and / or of different types.

[0062] The SENS device is configured to acquire (i.e. capture) an image sequence IMG_SEQ. This SENS device comprises one or more optical sensors, such as a camera, or an infrared sensor. As illustrated in Figure 1, the SENS device is configured to provide the image sequence IMG_SEQ to the APP device.

[0063] The IMG_SEQ image sequence comprises a plurality of consecutive images covering a portion of the sky. Each image in the IMG_SEQ sequence is associated with an acquisition time. Typically, within the images in the IMG_SEQ sequence, there are satellites orbiting the Earth, stars (in large numbers compared to the satellites), and measurements from noise (e.g. hot pixels).

[0064] It should also be noted that the satellites observed in the IMG_SEQ image sequence may include fixed geostationary satellites, drifting geostationary satellites, and orbiting satellites. In other words, the observed satellites may have geosynchronous orbits (GEO), medium Earth orbits (MEO), or low Earth orbits (LEO).

[0065] The APP device is configured to track the satellites observed in the image sequence IMG_SEQ. In other words, it is configured to determine the respective TRJ trajectories of the different satellites present in the image sequence IMG_SEQ.

[0066] In the context of the invention, a "satellite trajectory" is defined by a set of measurements associated with the satellite, each of these measurements comprising a position of the satellite (e.g. x and y coordinates of the satellite in the image), and a temporal index (e.g. index of the image in the sequence, or instant of acquisition of the image). For example, a satellite trajectory T m can be defined as follows: T m ={(x n , y n , i n ) I ne N m )}, with N m the indices of the measures associated with this trajectory.

[0067] According to the embodiment illustrated by Figure 1, the APP device is configured to provide the TRJ satellite trajectories to the OUT device. For example, the OUT device may be a rendering device comprising a screen for displaying the TRJ satellite trajectories. Alternatively, the OUT device could be a storage device used to record the TRJ satellite trajectories.

[0068] The APP device comprises, according to the embodiment illustrated by FIG. 1: at least one processing unit or processor PROC; and at least one memory MEM.

[0069] More particularly, the APP device has, according to one embodiment, the hardware architecture of a computer. The memory MEM constitutes an information medium in accordance with the invention, i.e. readable by the processor PROC and on which a computer program PROG in accordance with the invention is recorded. The program PROG comprises instructions for carrying out steps of a satellite tracking method in accordance with the invention, when the program PROG is executed by the processor PROC. The program PROG defines in particular the functional modules of the APP device represented in FIG. 2, which rely on or control the hardware elements of the latter.

[0070] As illustrated in Figure 1, the APP device has a COM communication module configured to communicate with the SENS acquisition device and / or the OUT device. No limitation is attached to the nature of the communication interfaces between these devices, which may be wired or wireless, and may implement any protocol known to those skilled in the art.

[0071] Having introduced the architecture of the APP satellite tracking device, we describe its operation below with reference to the following figures.

[0072] Figure 2 and Figure 3 respectively represent an example of architecture of a satellite tracking device and steps of a satellite tracking method according to an embodiment of the invention. These figures present the operation of the APP device introduced with reference to Figure 1.

[0073] The various functional modules of the APP device are illustrated in Figure 2. We describe the functions of these modules in relation to the process steps in Figure 3.

[0074] As illustrated by Figure 3, the proposed method comprises all or part of the steps S100 to S600 described below, these being implemented by the APP device.

[0075] The proposed method may comprise a plurality of iterations of each of its steps. The steps of the method may be repeated at a given frequency, for example every night, or every week.

[0076] In step S100, the obtaining module OBT of the APP device obtains the image sequence IMG_SEQ. From this, it obtains a set of measurements Si N associated with the spatial objects observed in the images of the IMG_SEQ sequence.

[0077] The APP device receives the image sequence IMG_SEQ from the SENS acquisition device, via its communication module COM. Alternatively, the APP monitoring device could also read the image sequence IMG_SEQ from a memory shared with the SENS acquisition device.

[0078] Then, the APP device processes the image sequence IMG_SEQ to obtain the measurement set Si N associated with the different spatial objects observed in the IMG_SEQ sequence. To do this, the APP device can use any known technique for detecting points in images. For example, it can use a high-pass filter, or a neural network.

[0079] For each spatial object observed in an image of the IMG_SEQ sequence, the APP device obtains a measurement (i.e. a point) in three dimensions. Each measurement includes x and y coordinates describing the position of the object in the image, and a temporal index (e.g. the index of the image in the sequence, the instant of image acquisition). Thus, the set of measurements SiN={(x n , y n , i n ) I ne N IN} with N IN the indices of the measurements obtained.

[0080] In step S200, the FLT filter of the APP device filters the measurement set Si N to eliminate measurements associated with stars. The APP device thus obtains a filtered set of measurements SFLT={(x n , y n , i n ) I ne N F LT} with N F T the indices of the preserved measures. Thus, the filtered set of measures S F LT does not include measurements considered to correspond to stars.

[0081] We describe in more detail the implementation of step S200 with reference to FIG. 4.

[0082] In step S300, the DET detector of the APP device detects, from the filtered measurement set SFLT, the satellite trajectories TRJ in the image sequence IMG_SEQ. The DET detector comprises (i.e. implements) at least one artificial neural network.

[0083] In step S400, the CLA classifier of the APP device classifies the detected satellite trajectories TRJ. The CLA classifier comprises (i.e. implements) at least one artificial neural network.

[0084] Following steps S300 and S400, the APP device obtains trajectories of satellites TRJ. As mentioned previously, the trajectory of a satellite T m is defined by a set of measures T m ={(x n , y n , i n ) I ne N m} with N mthe trajectory measurement indices. In addition, the APP device assigns a class C m to this trajectory T m .

[0085] The classes assigned to trajectories by the APP device indicate the types of satellite trajectories (i.e. orbits). For example, the list of assignable classes may include: fixed geostationary trajectory; drifting geostationary trajectory; moving trajectory (i.e. satellite in medium or low Earth orbit moving at high speed); and trailing trajectory (i.e. satellite in low Earth orbit moving at very high speed); and noise.

[0086] The implementation of steps S300 and S400 is described in more detail with reference to Figures 5 and 7. More specifically, Figure 5 illustrates a first embodiment of steps S300 and S400, while Figure 7 illustrates a second embodiment of these steps.

[0087] In step S500, the processing module PP of the APP device performs so-called post-processing (analytical) operations of the detected satellite trajectories TRJ.

[0088] This step makes it possible in particular to improve the results of the proposed satellite tracking solution in an image sequence IMG_SEQ. The implementation of step S500 is detailed below with reference to Figure 8.

[0089] In step S600, and according to one embodiment, the APP device provides the satellite trajectories TRJ to the OUT device, via its communication module COM.

[0090] As previously stated, each of the TRJ satellite trajectories provided by the APP device includes: a set of T measurements m describing the positions of a satellite in the images of the sequence IMG_SEQ; and a class C m indicating the type of satellite trajectory.

[0091] The OUT device can, for example, display the satellite trajectories TRJ on a screen. Alternatively, this OUT device can be used to store the satellite trajectories TRJ. These TRJ trajectories can be further analyzed by a separate device to catalog the orbits of the observed satellites.

[0092] We have presented here the general operation of the APP device proposed for tracking satellites in an image sequence IMG_SEQ. Also, we detail below the implementation of steps S200 to S500 with reference to the following figures.

[0093] Figure 4 represents steps of a method according to an embodiment of the invention for eliminating stars in a sequence of images. This figure details the implementation of the filtering step S200 to eliminate the measurements associated with stars.

[0094] As illustrated by FIG. 4, step S200 comprises all or part of steps S210 to S240 described below and implemented by the FLT filter of the APP device.

[0095] In step S210, the APP device selects, from the set of measurements Si N , the measurements whose brightness is greater than a threshold in the images of the IMG_SEQ sequence. Thus, the APP device selects the brightest points in the images of the IMG_SEQ sequence.

[0096] The inventors found that the majority of the brightest measurements in an image sequence are stars. For this reason, the APP device uses the brightest measurements to determine star motion information (either a star motion speed per image or a star rotation angle per image).

[0097] In step S220, the APP device determines the star displacement information from selected measurements.

[0098] First, the APP device determines whether the star paths are substantially straight in the image sequence IMG_SEQ, or whether they are substantially circular. To do this, it can determine whether the normals of the star paths intersect near an observation center (i.e., whether the intersection of these normals is close to the observation center).

[0099] In fact, if the image sequence IMG_SEQ is acquired by an observatory located near the equator, the trajectories of the stars are approximately rectilinear. The APP device then determines the average speed of the stars per image. In other words, it determines the displacement vector of the stars from one image to the next (e.g. -5.1 pixels / image along the axis, and -1.2 pixels / image along the y axis). To determine this displacement vector, the APP device calculates the average of the distances between each of the measurements and its nearest neighbor in the following image.

[0100] On the other hand, if the image sequence IMG_SEQ is acquired by an observatory located near the poles, the trajectories of the stars are approximately circular. In this case, the APP device determines the center of rotation of the stars (i.e. the intersection of the normals), as well as the rotation angle of the stars from one image to the next (e.g. 3.2 degrees / image).

[0101] In step S230, the APP device eliminates, using the star displacement information, the measurements considered to come from stars.

[0102] According to one embodiment, the APP device performs step S230 in the following manner. For each observed spatial object, the APP device predicts its positions in the other images of the sequence by assuming that this object is a star. If these predictions prove to be correct, then the APP device considers that the object is a star and eliminates the associated measurement.

[0103] More precisely, for each measurement in each image of the sequence, the APP device predicts the positions of the object in the other images of the sequence using the star displacement information. If the number of predicted positions considered correct is greater than a threshold, the APP device eliminates the considered measurement. A predicted position is considered correct when there is a measurement present in the image and whose distance to the predicted position is less than a threshold.

[0104] In step S240, the APP device eliminates, from among the remaining measurements, the isolated measurements in the image sequence IMG_SEQ.

[0105] Specifically, the APP device identifies measurements that do not have any near neighbors, and eliminates these measurements. A measurement is considered to have no near neighbors when there are no measurements located near it, i.e., no measurements below a distance threshold. The APP device considers in particular the image in which the measurement is located, but also the previous and following images (e.g., the two previous images, and the two following images).

[0106] According to one embodiment, the APP device can also eliminate measurements located on the edges (i.e. included below a given distance from the edges of the images) in the first images or the last images of the IMG_SEQ sequence.

[0107] Following these steps, the APP device obtains a filtered set of measurements S F LT, in which measurements considered to be associated with stars have been eliminated.

[0108] This set S F LT is then used by the APP device to detect satellite trajectories TRJ in the image sequence IMG_SEQ. In particular, we describe with reference to Figures 5 and 7 two embodiments of satellite trajectories detection and classification.

[0109] Figure 5 represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention. This figure illustrates a first embodiment of the DET detector and the CLA classifier, previously introduced with reference to Figures 2 and 3.

[0110] As mentioned previously, during step S300, the detector DET detects satellite trajectories TRJ in the image sequence IMG_SEQ from the filtered measurement set S FL T. Then, during step S400, the CLA classifier classifies (i.e. segments) the TRJ trajectories. [YES] This first embodiment proposes to use convolutional neural networks to implement the DET detector, and to use PointNet type neural networks to implement the CLA classifier.

[0112] The DET detector comprises, according to this first embodiment, several CNN convolutional neural networks LS , CNN H s, and CNN V HS- Each of these networks takes as input an image in which the set of filtered measurements S is superimposed F LT, and outputs BB frames encompassing detected satellite trajectories (i.e. "bounding boxes"). Such a B frame m can for example be defined as follows: B m ={x m , y m , h m , L m} with x m and there m the coordinates of the center of the frame, h m its height and m its width.

[0113] CNN neural networks LS , CNN HS , and CNN VH are respectively dedicated (i.e. specialized, trained) to the detection of satellite trajectories of different speeds. In particular, the CNN network LS is trained to detect satellite trajectories in geostationary orbit that are substantially fixed or drifting (i.e. low-speed trajectories). The CNN network H is trained to detect trajectories of satellites in medium or low orbit and moving at high speed (i.e. scrolling type trajectories). Finally, the CNNVHS network is trained to detect trajectories of satellites in low orbit and moving at very high speed (i.e. trail type trajectories).

[0114] The CNN network Ls can use a Faster R-CNN type neural network with an Inception V2 type feature extraction network (i.e. "backbone network"). Faster R-CNN type networks and their implementation are for example described in the following document: Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", 2016. And, Inception V2 type networks and their implementation are for example described in the following document: Szegedy et al., "Rethinking the Inception Architecture for Computer Vision", 2015.

[0115] In addition, CNN networks H s and CNN V HS can respectively use a Faster R-CNN type neural network with a Resnet 101 type feature extraction network. ResNet 101 type networks and their implementation are for example described in the following document: He et al., “Deep Residual Learning for Image Recognition”, 2015.

[0116] Training of the DET detector's neural networks is performed using simulated training data, i.e., images comprising computer-generated satellite trajectories to which noise is added. Neural networks can also be pre-trained using public datasets, such as the "COCO" dataset.

[0117] More specifically, to train the neural networks of the proposed solution, it can be considered to generate data including star trajectories (circular or rectilinear), trajectories of space objects of different types, and noise. Then, the measurements are filtered to eliminate stars using the technique previously described in reference in Figure 3. Finally, the measurements thus obtained are used to train the neural networks.

[0118] To dedicate the CNN network LSto the detection of satellite trajectories in geostationary orbit that are substantially fixed or drifting (i.e. low-speed trajectories), it is trained using only these types of trajectories. Similarly, the CNN network H is trained using scrolling type trajectories, and the CNNVHS network is trained using trailing type trajectories.

[0119] For implementation details regarding the training of neural networks, including loss functions, those skilled in the art may refer to the aforementioned documents and the references cited in these documents.

[0120] According to a particular embodiment, the CNNLS neural network processes in parts the image in which the set of filtered measurements S is superimposed. FL T. In contrast, the CNNHS and CNNVHS networks take this entire image as input.

[0121] The CLA classifier comprises, according to this first embodiment, several PN neural networks L s and PN HS PointNet type. These PN networks L s and PN HS are respectively dedicated to the classification (i.e. segmentation) of trajectories of TRJ satellites of different speeds.

[0122] On the one hand, the PN network LS is trained to classify satellite trajectories in geostationary orbit as substantially fixed or drifting (i.e. low-speed trajectories). As illustrated in Figure 7, the PN network LS takes as input the BB frames provided by the CNN network LS , dedicated to the detection of low-speed trajectories.

[0123] The PN network LS treats each of the BB frames separately. For each B frame m encompassing a trajectory T m , it takes as input all the measurements located inside frame B min the image sequence IMG_SEQ. Note also that the measurements located in frame B m are refocused before providing them to the PN network LS so that the x coordinates m and there m from the center of the frame are equal to 0.

[0124] The PN network LS provides a label as output for each of the measures located in frame B m , this label indicating whether or not the measurement belongs to the trajectory T m . Thus, the PN network L s allows to separate the measurements belonging to the TRJ trajectories from the noise. In addition, it provides a class C output m assigned to trajectory T m .

[0125] The list of classes attributable by the PN network LS is: fixed geostationary trajectory; drifting geostationary trajectory; double drifting geostationary trajectories (two drifting trajectories are present in the same frame); and noise.

[0126] On the other hand, the PN network HS is trained to classify satellite trajectories in medium or low orbit (i.e. high-speed trajectories). As illustrated in Figure 1, the PN network H S takes as input the BB frames provided by the CNN networks HS and CNN VHS , dedicated to the detection of high-speed trajectories.

[0127] The PN network H each of the BB frames is treated separately. For the PN network H s, only measurements around the diagonals of the BB bounding boxes are provided (i.e. included below a threshold distance from the diagonals). These measurements are recentered before providing them to the PN network H s so that the x coordinates m and there m from the center of the frame are equal to 0. In addition, these measurements are normalized before providing them to the PN network Hs, and this by dividing each of the coordinates of the measurements by the maximum absolute value of the coordinates in x and y.

[0128] The PN network HS provides a label as output for each of the measures located in frame B m , this label indicating whether or not the measurement belongs to the trajectory T m . It also provides a class C output m assigned to trajectory T m .

[0129] The list of classes attributable by the PN network HS is: high-speed trajectory (scrolling-type trajectory or trailing-type trajectory); and noise. Then, the APP device can distinguish between scrolling-type trajectories and trailing-type trajectories as follows. Scrolling-type trajectories comprise on average less than 1.5 measurements per frame, while trailing-type trajectories comprise on average more than 1.5 measurements per frame.

[0130] For each bounding box Bm , the APP device thus obtains: a set of measurements associated with the trajectory T m ={(x n , y n , i n ) I ne N m} with N m the indices of the trajectory measurements; and a class C m assigned to trajectory T m .

[0131] PointNet networks and their implementation are for example described in the previously cited document Qi et al. PN networks LS and PN H PointNet-type s are trained using simulated data.

[0132] To dedicate the PN network L s to the classification (segmentation) of satellite trajectories in geostationary orbit that are substantially fixed or drifting (i.e. low-speed trajectories), it is trained using only these types of trajectories. Similarly, the PN network His trained using scrolling type trajectories and trailing type trajectories (i.e. high speed trajectories).

[0133] For illustration purposes, the following figure represents the data processed and obtained by the different modules of the APP device according to this first embodiment.

[0134] Figure 6 represents examples of images processed by a satellite tracking device according to one embodiment of the invention.

[0135] The first image IMG_SUP represents a superposition of the images of the sequence IMG_SEQ obtained by the obtaining module OBT during step S100.

[0136] The second image IMG_SUP_FLT represents within the same image the filtered set of measurements S F LT obtained by the FLT filter following step S200.

[0137] The third image IMG_TRJ_DET represents within the same image the BB frames encompassing the satellite trajectories detected by the DET detector following step S300.

[0138] The fourth image IMG_TRJ_CLA represents within the same image the trajectories of TRJ satellites obtained by the CLA classifier following step S400.

[0139] The fifth image IMG_TRJ_PP represents within the same image the trajectories of TRJ satellites obtained by the PP processing module following step S500.

[0140] We have described above a first embodiment of the DET detector and the CLA classifier of the APP device. However, within the scope of the invention, other embodiments may be envisaged. Also, we describe below with reference to Figure 7 a second embodiment of the detection and classification of satellite trajectories.

[0141] Figure 7 represents an example of architecture of a satellite tracking device in a sequence of images according to an embodiment of the invention. This figure illustrates a second embodiment of the DET detector and the CLA classifier, previously introduced with reference to Figures 2 and 3.

[0142] The DET detector comprises, according to this second embodiment, at least one Transformer-type TRF neural network, i.e. a network comprising an attention mechanism. Transformer-type TRF neural networks and their implementation are for example described in the document A. Vaswani et al. cited above.

[0143] According to this second embodiment, the TRF network of the DET detector takes as input the measurements of the set S FLT, and provides a latent vector LV for each of these measurements. Then, a CLST association module of the DET detector associates (i.e. groups) the measurements belonging to the same satellite trajectory T m as a function of the Euclidean distances between the latent vectors LV.

[0144] The APP device thus detects the respective TRJ trajectories of the different satellites observed in the IMG_SEQ image sequence. More precisely, for each detected trajectory T m , the APP device obtains a set of measurements associated with the trajectory T m ={(x n , y n , i n ) | ne N m} with N m the indices of the trajectory measurements.

[0145] For example, the CLST association module uses a technique called agglomerative hierarchical clustering with a defined distance threshold. The details of implementing the agglomerative hierarchical clustering technique are known to those skilled in the art.

[0146] However, within the framework of the invention, embodiments could be envisaged in which the CLST association module uses other grouping algorithms to associate the measurements belonging to the same satellite trajectory from the distances between the latent vectors.

[0147] The CLA classifier comprises, according to this second embodiment, at least one CN classification neural network. This CN neural network processes the different detected satellite trajectories TRJ separately. For each trajectory T m , the CN classification network takes as input the measurements associated with this trajectory T m , and outputs a class C m assigned to this trajectory T m .

[0148] There are no limitations attached to the architecture of the CN classification neural network. For example, the CN network can be a multi-layer perceptron, comprising several dense layers with ReLU-type activation functions, and an output layer with a Softmax-type activation function.

[0149] It should be noted that TRF and CN neural networks can be trained from simulated (i.e. computer-generated) training data.

[0150] Within the scope of the invention, embodiments could also be envisaged in which the DET detector comprises a plurality of Transformer-type neural networks respectively dedicated to the detection of trajectories of TRJ satellites of different speeds. Similarly, the CLA classifier could comprise a plurality of classification neural networks respectively dedicated to the classification of trajectories of TRJ satellites of different speeds.

[0151] Following step S300 of detection and step S400 of classification of the satellite trajectories TRJ, the APP device performs during step S500 various post-processing operations of these trajectories TRJ. As mentioned previously, step S500 makes it possible to improve the reliability of the satellite tracking carried out by the APP device. We detail the implementation of this step with reference to the following figure.

[0152] Figure 8 represents steps of a method according to an embodiment of the invention for carrying out post-processing operations on satellite trajectories. This figure details the implementation of step S500 introduced with reference to Figures 2 and 3.

[0153] As illustrated by Figure 8, and according to one embodiment, step S500 comprises all or part of steps S510 to S550 described below and implemented by the processing module PP of the APP device.

[0154] In step S510, the APP device merges (i.e. combines) satellite trajectories TRJ to group the measurements associated with several detected satellite trajectories TRJ and belonging to the same satellite. Thus, if measurements belonging to the same satellite have been associated with several trajectories by the APP device, then these measurements are grouped within the same trajectory.

[0155] To do this, the APP device compares the trajectories of TRJ satellites two by two, and merges them or not depending on the result of the comparison.

[0156] Specifically, the APP device compares the number of measurements common to two trajectories T m and T p . If this number is greater than a given percentage (eg 20%) of the number of measurements of the smallest of the two trajectories, then the measurements of the two trajectories T m And T p are grouped within a single trajectory. In other words, the APP device then achieves the union of the two trajectories T m and T p .

[0157] It is important to note that the APP device does not compare all types of trajectories together. On the one hand, fixed geostationary trajectories are compared only with each other. On the other hand, drifting geostationary trajectories, moving geostationary trajectories, and trailing geostationary trajectories are compared together.

[0158] In step S520, the APP device performs a completion of the TRJ satellite trajectories of scrolling types, i.e., high-speed trajectories comprising on average less than 1.5 measurements per image. This completion uses three-dimensional linear regressions.

[0159] For a trajectory T m of a satellite, the completion carried out by the APP device makes it possible to eliminate measurements associated with the trajectory T m, but not belonging to this satellite. In addition, this completion allows the addition of measurements belonging to the satellite and which would not have been associated with the trajectory T m .

[0160] For each measurement of the trajectory T m , the APP device predicts the position of the object using three-dimensional linear regression determined from the measurements of the trajectory T m . If there is another measurement closer to the predicted position than the measurement considered, then the measurement considered is eliminated from the trajectory T m , and is replaced by the other, closer measure.

[0161] For each image i k of the IMG_SEQ sequence, the APP device proceeds as follows. If the trajectory T m includes a measurement on this image i k, and if the distance between this measurement and a predicted position is greater than a threshold, then the measurement is eliminated from the trajectory T m . If the trajectory T m does not include a measurement in this image i k , and if the distance between the predicted position and the measurement closest to the predicted position is less than a threshold, then this measurement is associated (i.e. added) to the trajectory T m .

[0162] In step S530, the APP device filters out trail-type TRJ trajectories, i.e., high-speed trajectories comprising on average more than 1.5 measurements per image.

[0163] For each trajectory T m of the drag type, the APP device determines the slope and the ordinate at the origin of the linear regression line formed by the measurements of the trajectory T m . It then eliminates, among the measurements of the trajectory T m, those which deviate beyond a given distance from the linear regression line.

[0164] The APP device thus eliminates measurements that are not aligned along the two-dimensional trajectory. In this sense, it performs a polishing of the trail-type trajectories.

[0165] In step S540, the APP device merges the trail-type trajectories. This makes it possible to prevent the same trail-type trajectory from being detected in several disjoint pieces by the APP device. To this end, the APP device compares the trail-type trajectories with each other, and merges them or not depending on the result of the comparison.

[0166] In fact, the device compares two by two the linear regression lines of the trail-type trajectories. For two trajectories T m and T pof the trail type, if the difference between their slopes is less than a threshold, and if the difference between their ordinates at the origin is less than another threshold, then the APP device groups the measurements of the two trajectories T m and T p within a single trajectory.

[0167] In step S550, the APP device filters the satellite trajectories TRJ so that each of the trajectories comprises only one measurement per image of the sequence IMG_SEQ.

[0168] If a trajectory T m includes several measurements on an image of the sequence IMG_SEQ, the APP device combines these measurements to obtain a single measurement. For example, the position of the single measurement obtained by the APP device may correspond to the barycenter of the positions of the measurements of the trajectory T m included in this image.

[0169] Additional variants: It should be noted that the order in which the steps of a method according to the invention are carried out, in particular with reference to the attached drawings, constitutes only an example of embodiment without any limiting character, variants being possible. In particular, a method according to the invention may comprise one or more iterations of the steps described above, in particular with reference to the attached drawings. Furthermore, the reference signs do not limit the scope of the protection, their sole function being to facilitate the understanding of the claims.

[0170] A person skilled in the art will understand that the embodiments and variants described above constitute only non-limiting examples of implementation of the invention. In particular, a person skilled in the art may envisage any adaptation or combination of the embodiments and variants described above in order to meet a very specific need.

Claims

Claims 1. Method for tracking satellites in an image sequence (IMG_SEQ), the method comprising: a step of obtaining (S100) a set of measurements (Si N ) associated with spatial objects observed in the images of the sequence (IMG_SEQ), each measurement comprising a position of an object in an image and a temporal index of the image; a step of detecting (S300) satellite trajectories (TRJ) in the sequence of images (IMG_SEQ), by a detector (DET) comprising at least one neural network and using all or part of the set of measurements obtained (Si N ); and a step of classification (S400) of the detected satellite trajectories (TRJ), by a classifier (CLA) comprising at least one other neural network.

2. Method according to claim 1, in which: the detector (DET) comprises at least one convolutional neural network (CNN), and determines (S200) frames (BB) each encompassing a satellite trajectory (TRJ) in the images of the sequence (IMG_SEQ); and the classifier (CLA) comprises at least one PointNet type neural network, and for each frame (BB) encompassing a satellite trajectory (TRJ), the classifier (CLA) determines (S300) the measurements located in the frame belonging to this trajectory, and assigns (S300) a class to this trajectory.

3. Method according to claim 1, in which the detector (DET) comprises at least one Transformer type neural network (TRF), and associates (S200) the measurements belonging to the same satellite trajectory (TRJ) in the images of the sequence (IMG_SEQ).

4. Method according to claim 3, in which the detector (DET) determines (S200) latent vectors (LV) representative of said measurements (SF LT) using said at least one Transformer-type neural network, and associates (S200) measurements with the same satellite trajectory (TRJ) as a function of distances between the latent vectors (LV).

5. Method according to any one of claims 1 to 4, wherein: the detector (DET) comprises a plurality of neural networks (CNN LS , CNN HS , CNN VH s) respectively dedicated to the detection of satellite trajectories (TRJ) of different speeds; and the classifier (CLA) comprises a plurality of neural networks (PN LS , PN H s) respectively dedicated to the classification of satellite trajectories (TRJ) of different speeds.

6. Method according to any one of claims 1 to 5, comprising a step of filtering (S200) the set of measurements (Si N) to eliminate measurements associated with stars, and in which the detector (DET) uses only the filtered set of measurements (S FL T) to carry out the step of detecting (S300) satellite trajectories (TRJ) in the image sequence (IMG_SEQ).

7. Method according to claim 6, in which the step of filtering (S200) the set of measurements (SIN) to eliminate the measurements associated with stars comprises: a determination (S220) of displacement information of the stars from measurements whose brightness is greater than a threshold in the images of the sequence (IMG_SEQ); and a step of eliminating (S230) the measurements considered to be associated with stars from the determined displacement information.

8. Method according to any one of claims 1 to 7, further comprising: one or more steps of merging (S510, S540) the satellite trajectories to group the measurements associated with several detected satellite trajectories (TRJ) and belonging to the same satellite; and a step of filtering (S550) the satellite trajectories (TRJ) so that a said satellite trajectory (TRJ) comprises only one measurement per image of the sequence (IMG_SEQ).

9. Device (APP) for tracking satellites in a sequence of images (IMG_SEQ), the device (APP) comprising: an obtaining module (OBT) configured to obtain (S100) a set of measurements (S !N) associated with spatial objects observed in the images of the sequence (IMG_SEQ), each measurement comprising a position of an object in an image and a temporal index of the image; a detector (DET) configured to detect (S300) satellite trajectories (TRJ) in the sequence of images (IMG_SEQ), the detector (DET) comprising at least one neural network and using all or part of the set of measurements obtained (Si N ); and a classifier (CLA) configured to classify (S400) the detected satellite trajectories (TRJ), the classifier (CLA) comprising at least one other neural network.

10. Surveillance system (SYS) comprising: a device (APP) for tracking satellites according to claim 9; and a device (SENS) for acquiring a sequence of images (IMG_SEQ).

11. Computer program (PROG) comprising instructions for implementing the steps (S100-S600) of a method according to one of claims 1 to 8, when said computer program (PROG) is executed by at least one processor (PROC).

12. Computer-readable information medium (MEM) comprising a computer program (PROG) according to claim 11.

Images