Computer system and method for training and using image segmentation model to detect resident space object, for generating simulated training image therefor, and for detecting and tracking resident space object

Abstract

To provide a method of generating a simulated image for training a model to detect resident space objects (RSOs), a method of training the model, and a method of detecting RSOs.SOLUTION: A method includes generating a foreground of a simulated image including RSOs by calculating coordinates of an imaging satellite at a given time, determining an area of interest given a field of view (FOV) of the imaging satellite, and calculating coordinates of all RSOs and saving the coordinates if the RSO is within the area of interest. The method further includes generating a background of the simulated image including stars by querying a star catalogue database for coordinates of all stars that fall inside the FOV, adding noise, and translating RSO and star coordinates to pixel coordinates. The method further includes generating the simulated image by choosing an imaging mode, setting an exposure time, and rendering the simulated image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The following generally relates to space situational awareness (SSA), and more particularly to systems and methods for detecting, classifying, and correlating resident space objects (RSOs) from an optical payload platform in space or on the ground.

Background Art

[0002] For future spacecraft navigation needs, it is becoming increasingly important to detect and track RSOs using on-orbit payloads. Conventional methods are slow and often require human operators to analyze downlinked data or external data for SSA and subsequently task ground or space assets to provide more observations to improve the tracking accuracy of RSOs. The ability to perform this task autonomously may reduce the response time of ground-based decision support systems and may also enable more detections to improve overall RSO tracking accuracy. As the number of objects in orbit increases, the risk from decision support systems used to avoid unplanned collisions also increases. The disadvantages associated with conventional SSA mean that the probability of a collision with invisible debris or obstacles in the probabilistic assessment of a collision can continue to increase with the trends in the space environment.

[0003] For future spacecraft navigation, it is further desirable to detect, track, and classify threats of the rendezvous-proximity operator type. For future spacecraft navigation, a rapid sensor cuing response from satellite to satellite is further desirable.

[0004] Therefore, there is a need for improved systems and methods for detecting and tracking RSOs that overcome at least some of the drawbacks of existing systems and methods.

Summary of the Invention

Means for Solving the Problem

[0005] A method for generating simulated images for training an image segmentation model to detect resident space objects (RSOs) is provided. The method includes calculating the coordinates of an imaging satellite at a given time, determining a region of interest given the field of view (FOV) of the imaging satellite, and calculating the coordinates of all RSOs and saving the coordinates if the RSO is within the region of interest, thereby generating the foreground of the simulated image, where the foreground includes RSOs; querying the star catalog database for the coordinates of all stars that fall within the FOV; adding noise to the simulated image; and generating the background of the simulated image by converting the RSO coordinates and star coordinates to pixel coordinates in the simulated image, where the background includes stars; selecting an imaging mode, setting an exposure time, and generating the simulated image by rendering the simulated image according to the imaging mode and exposure time.

[0006] Calculating the coordinates of all RSOs may be performed using a propagation algorithm.

[0007] The noise may include star streaks.

[0008] The imaging mode may be selected from among stellar tracking, target rate tracking, and custom rate.

[0009] Calculating the coordinates of all RSOs and saving the coordinates if the RSO is within the region of interest may include simulating the orbital path to generate a list of access windows when the foreground satellite is within the FOV.

[0010] The time stamp associated with each access window may be slightly randomized so that the foreground satellite does not always have to be centered in the simulated image.

[0011] The timestamps associated with each access window may be used to calculate the right ascension (RA) and declination (DEC) of all foreground satellites within the FOV.

[0012] Converting the RSO to pixel coordinates in the simulated image may be performed according to the RA and DEC.

[0013] A method for training an image segmentation model to detect resident space objects (RSOs) is provided, the method including generating a training set by executing a method for generating simulated images for a plurality of simulated images, and training the image segmentation model using the training set and / or one or more real images.

[0014] The method may be executed on a low-power GPU or FPGA.

[0015] A method for detecting a resident space object (RSO) in an optical image is provided, the method including inputting the optical image into a trained image segmentation model trained using a method for training the image segmentation model, and detecting the RSO in the optical image using the trained image segmentation model.

[0016] A method for training an image segmentation model for detecting RSOs is provided. The method includes generating a simulated training dataset, where the simulated training dataset includes at least one simulated training image, and each simulated training image simulates the field of view from an imaging satellite or a ground-based optical system; customizing the simulated training dataset for edge cases, different sensor types, and / or different imaging satellites, and curating the simulated training dataset by performing one or more of combining different simulated training datasets; processing the simulated training dataset into a training subset, a test subset, and a validation subset; generating an RSO detection model by training a modified U-Net on the training subset; inspecting the generated RSO detection model using the test subset; and validating the generated RSO detection model by inputting the validation subset into a plate-solving algorithm.

[0017] A method for detecting RSOs in optical images is provided. The method includes generating simulated training images using an image simulator, where each of the simulated training images simulates the field of view from an imaging satellite or a ground-based optical sensor and includes a simulated foreground including an RSO and a simulated background including stars; training an RSO detection model using the simulated training images; configuring the trained RSO detection model for on-board processing, where the on-board processing is carried on a satellite; and detecting RSOs in optical images using the trained RSO detection model configured for on-board processing.

[0018] The image simulator may apply a configurable scale of brightness that incorporates pseudo-random elements and generates realistic brightness variations across simulated training images. The size of the RSO in the simulated foreground may be randomized between 3 and 5 pixels.

[0019] A method for detecting RSOs mounted on a satellite is provided. The method includes capturing an optical image using an optical sensor mounted on the satellite and providing the optical image to a processor mounted on the satellite. The processor executes an RSO detection model trained using a plurality of simulated training images that simulate the field of view from the imaging satellite. Each simulated training image includes a simulated foreground containing an RSO and a simulated background containing stars. The method further includes detecting the RSO in the optical image using the RSO detection model and controlling the attitude of the satellite based on the output of the RSO detection model.

[0020] The method may be executed on a low-power GPU or FPGA.

[0021] A method for generating simulated images for training an image segmentation model to detect resident space objects (RSOs) is provided. The method includes generating a foreground for the simulated image by calculating the coordinates of the imaging satellite at a given time, determining a region of interest given the field of view (FOV) of the imaging satellite, and calculating the coordinates of all RSOs and saving the coordinates if the RSO is within the region of interest. The foreground includes the RSO. The method further includes generating a background for the simulated image by querying a star catalog database for the coordinates of all stars that fall within the FOV, adding noise to the simulated image, and converting the RSO coordinates and star coordinates to pixel coordinates in the simulated image. The background includes stars. The method further includes generating the simulated image by selecting an imaging mode, setting an exposure time, and rendering the simulated image according to the imaging mode and exposure time.

[0022] A method for training an image segmentation model to detect resident space objects (RSOs) is provided, the method including generating a training set by performing, for a plurality of simulated images, generating the simulated images as described above, and training an image segmentation model using the training set.

[0023] A method for detecting resident space objects (RSOs) in optical images is provided, the method including inputting an optical image into a trained image segmentation model trained as described above, and detecting RSOs in the optical image using the trained image segmentation model.

[0024] A method for training an image segmentation model for detecting RSOs is provided, the method including generating a simulated training dataset, the simulated training dataset including at least one simulated training image, each simulated training image simulating a field of view from an imaging satellite or a ground-based optical system. The method further includes curating the simulated training dataset by performing one or more of customizing the simulated training dataset for edge cases, different sensor types, and / or different imaging satellites, and combining different simulated training datasets. The method further includes processing the simulated training dataset into a training subset, a test subset, and a validation subset. The method further includes generating an RSO detection model by training a modified U-Net convolutional neural network (CNN) against the training subset. The method further includes inspecting the generated RSO detection model using the test subset. The method further includes validating the generated RSO detection model by inputting the validation subset into a plate solving algorithm.

[0025] A method for detecting RSO in an optical image is provided, the method including generating simulated training images using an image simulator, each of the simulated training images simulating the field of view from an imaging satellite or a ground-based optical sensor and including a simulated foreground including RSO and a simulated background including stars. The method further includes training an RSO detection model using the simulated training images. The method further includes configuring the trained RSO detection model for on-board processing, the on-board processing being mounted on a satellite. The method further includes detecting RSO in the optical image using the trained RSO detection model configured for on-board processing.

[0026] A method for detecting RSO mounted on a satellite is provided, the method including capturing an optical image using an optical sensor mounted on the satellite and providing the optical image to a processor mounted on the satellite, the processor executing an RSO detection model trained using a plurality of simulated training images that simulate the field of view from an imaging satellite, each of the simulated training images including a simulated foreground including RSO and a simulated background including stars, detecting RSO in the optical image using the RSO detection model, and controlling the attitude of the satellite based on the output of the RSO detection model.

[0027] Upon consideration of the following description of some exemplary embodiments, other aspects and features will become apparent to those skilled in the art.

[0028] The drawings included herein are for the purpose of illustrating various examples of the articles, methods, and apparatuses herein.

Brief Description of the Drawings

[0029]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

DETAILED DESCRIPTION OF THE INVENTION

[0030] To provide an example of each claimed embodiment, various devices or processes are described below. The embodiments described below do not limit any claimed embodiment, and any claimed embodiment may cover a process or device different from those described below. The claimed embodiments are not limited to a device or process having all the features of any one device or process described below, or to features common to a plurality or all of the devices described below.

[0031] One or more systems described herein may be implemented by a computer program that executes on a programmable computer, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, a server, a personal computer, a single-board computer, a microcontroller, a cloud-based program or system, a laptop, a personal digital assistant, a cellular phone, a smartphone, or a tablet device.

[0032] Each program is preferably implemented in a high-level procedural or object-oriented programming language, and / or a scripting language for communicating with the computer system. However, the program may be implemented in assembly language or machine language if desired. In any case, the language may be a compiled language or an interpreted language. Each such computer program is preferably stored on a storage medium or device readable by a general-purpose or special-purpose programmable computer that configures and operates the computer when read by the computer to execute the procedures described herein.

[0033] The description of embodiments having several components communicating with each other does not imply that all such components are required. On the contrary, various optional components are described to show the various possible embodiments of the present invention.

[0034] Furthermore, process steps, method steps, algorithms, etc. may be described sequentially (in the present disclosure and / or in the claims), but such processes, methods, and algorithms may be configured to function in an alternative order. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of the processes described herein may be performed in any practical order. Further, some steps may be performed simultaneously.

[0035] When a single device or article is described herein, it will be readily apparent that two or more devices / articles may be used instead of the single device / article (regardless of whether they cooperate). Similarly, when two or more devices or articles are described herein (regardless of whether they cooperate), it will be readily apparent that a single device / article may be used instead of the two or more devices / articles.

[0036] The following generally relates to detecting resident space objects (RSOs) in any orbital regime, and more particularly to training a neural network to detect RSOs for space situational awareness (SSA) using space-based and ground-based optical payloads and low SWaP onboard processing (OBP), as well as detecting RSOs using the trained machine learning model. The orbital regime may include any one or more of low Earth orbit (LEO), medium Earth orbit (MEO), geostationary orbit (GEO), high Earth orbit (HEO), cislunar (between the Earth and the lunar orbit), and deep space.

[0037] Advances in machine learning (ML) enable spacecraft systems to detect and evaluate collision risks using onboard sensors. Instead of relying solely on ground-based manual evaluations of third-party data sources, satellites can perform their own collision risk identification and mitigation without human involvement, reducing the risk and cost of destructive collisions. Satellites may do so using a network of space situational awareness sensors. The systems and methods of the present disclosure may advantageously make the SSA sensor network smarter and improve the quality of SSA observations fed into the network.

[0038] To address the challenges of SSA in crowded orbits, the present disclosure provides an ML model trained to detect the location of RSOs in a star field using optical images. This is achieved in three parts: (1) creation of an image simulator for generating a large dataset of simulated satellite images, (2) creation and training of a modified U-Net CNN and the resulting model, and (3) deployment and optimization of the trained model on a low SWaP GPU.

[0039] (1) The training in (1) may be further performed using real images in addition to, or instead of, the simulated satellite images.

[0040] The ability to simulate different sensor types or proprietary image data is desired, and thus a fully configurable image simulator is provided by the present disclosure. The ability to simulate different imaging modes with different exposure times, including star tracking, target rate tracking, and custom rates, is further provided by the present disclosure. To simulate a particular camera, the user of the image simulator may configure attributes such as the field of view (FOV) and image size. For this attempt, if actual payload data was available for inspection, the simulator was configured to create a synthetic image representing a general satellite mission. Additionally, the image simulator characterizes various types of noise, dead pixels, and other common attributes found in on-orbit data to enable the user to emulate optical sensors on a variety of orbits and to enable the user to specify attributes in the image.

[0041] In one embodiment, when trained, the modified U-Net CNN model of the present disclosure detected the location of RSOs in the star field with an accuracy of 83%. The trained model was then deployed on an Nvidia Xavier Jetson AGX GPU. Optimization of throughput, power, efficiency, and inference time was achieved. Experiments were conducted using various inference engines. The trained model was able to successfully predict the location of RSOs from optical data images within 23 ms.

[0042] The amount of space debris and satellites in orbit is increasing, making the on-board detection of resident space objects (RSOs) even more important for spacecraft situational awareness and collision mitigation. This may involve using a space situational awareness (SSA) sensor network and collision mitigation. Low-cost on-board optical sensors are increasingly being used to produce image data for SSA. The ability to accurately interpret optical sensor image data for RSOs and rapidly propagate their orbits is a fundamental building block for applications such as automated flight dynamics for collision avoidance.

[0043] In one embodiment, the neural network-based model of the present disclosure extracts position information for RSOs in optical images of star streaks and is trained using simulated images based on existing space heritage, SSA optical payloads. The inspection of the model is based on the predicted proximity of the RSO to the true masked RSO position. The model was able to achieve an accuracy of 83% using fully simulated on-orbit data. Two-thirds of the detection errors are due to disappearing RSOs and one-third are due to false positives. Reducing the features associated with unclassified image data improves detection performance, suggesting that a more extensive on-orbit optical image data set and a larger temporal coverage will improve model accuracy. The trained model was then executed on a configurable graphical processing unit (GPU) in an attempt to emulate a space-qualified system-on-chip (SoC) and performed RSO location prediction within 23 ms.

[0044] The success of the trained model demonstrates that the present disclosure is suitable for closed-loop applications such as automated flight control and other time-constrained operations. This ability can prove to be key when integrated with an on-board SSA system. In some cases, a network of sensors may be used to predict impending collisions. Additionally, automated flight control may be slow and predictive warnings may be required to enable the flight control to react.

[0045] Other efforts in the space community have trained ML models that use existing available payload data using a ground-based infrastructure. The ability to perform closed-loop RSO detection, classification, orbit determination, and servicing on a payload would advantageously enable applications that require automated remote proximity operations (RPO), such as detecting, tracking, and classifying, tasking (tipping and cueing and optimization), and information production to support real-time concepts of operations for SSA.

[0046] Advantageously, compared to existing technologies, the present disclosure allows for the production of application-specific ML models to enhance the performance of the algorithms according to the characteristics required for a given application.

[0047] On-board detection of RSOs and classification of RSOs are important emerging strategic themes in the space industry and related industries. This ability to detect RSOs will be important for any automated flight dynamics, especially for next-generation space-based optical missions.

[0048] The present disclosure includes an image simulator for generating a large dataset of simulated satellite images.

[0049] The present disclosure further includes a neural network (preferably, a U-Net convolutional neural network (CNN) algorithm) that generates or outputs a model during training and / or modification. The neural network may be any suitable neural network, particularly any suitable CNN.

[0050] The present disclosure further includes the deployment and optimization of a trained model on a low-power GPU.

[0051] This disclosure may have further applications in the use of the above CNN for generating models and, thereby, in one example, for leveraging the above capabilities for an on-orbit decision-making algorithm, system, method, and / or device to find patterns or make decisions from datasets not previously seen in a broader SSA system.

[0052] While a CNN is described as a preferred embodiment of a neural network for training a model, it will be understood that other neural networks are explicitly contemplated in this disclosure.

[0053] Other technical apparatuses or software abstractions for training a model and / or for image segmentation are explicitly contemplated herein.

[0054] The system, method, and device for SSA OBP according to this disclosure may generate simulated images for training a CNN model, which may be a U-Net model, more efficiently (through fewer parameters) and more quickly (through the use of simulated images rather than waiting for a sufficient number of real images) than conventional systems, methods, and devices.

[0055] The deployment and optimization of such a trained U-Net model in a low-power GPU context may advantageously favor this system, method, and device over conventional systems, methods, and devices for SSA OBP.

[0056] Specifically, spacecraft, as well as their design, development, and deployment, are known to be extremely expensive endeavors, and any additional weight on a spacecraft can significantly increase the cost of launching the spacecraft.

[0057] Therefore, since high-performance computers increase the size, weight, and power (SWaP) of a spacecraft, it is desirable to keep the SWaP of the spacecraft as small as possible, which disadvantages the use of such high-performance computers.

[0058] Furthermore, since the power budget on a spacecraft can be extremely small, it is highly desirable to perform inference using a low-performance graphics processing unit (GPU) or a field-programmable gate array (FPGA) further provided by the present disclosure. Thus, the above model implemented on such a low-performance GPU can advantageously be executed very quickly on a low-power device or in a very low-power mode and thus can be feasible for space applications.

[0059] Therefore, deploying and optimizing the above-trained U-Net model in a low-power GPU context can present additional advantages from the perspectives of cost savings and feasibility.

[0060] The systems, methods, and devices for SSA OBP according to the present disclosure include a series of artificial intelligence software modules developed for the purpose of detecting, classifying, and correlating RSOs from an optical payload platform in space or on the ground.

[0061] The software modules utilize neural networks (such as CNNs) to optimize performance with respect to the application or the characteristics of the input data.

[0062] To support what the machine learning requirements do, a novel usage of an image simulator has been developed.

[0063] Space-based sensors are simulated, for example, with sensor noise, sky noise, light gradients, star streaks, and the introduction of a reference image of the RSO.

[0064] Artificial intelligence (AI) software is used for image interpretation and as input to flight dynamics, tasking, or processing subroutines. The above is optimized for space hardware.

[0065] Detecting, classifying, and maintaining the orbital information of RSOs is known to be an extremely difficult problem to solve. The generation and use of synthetic images to train an AI model to detect the location of RSOs in optical images, classify RSOs, and determine their orbits represents a significant advancement in the field of SSA. Moreover, the advantage of the above is the use of low-power GPUs or FPGAs to enable this new approach.

[0066] This disclosure includes tuning an AI classifier to function in a low SWaP computing environment. Conventional implementations of AI for SSA focus on the ground segment or high-performance GPUs that are not accessible for space missions for cost and power reasons.

[0067] The accuracy of the orbits calculated in the image simulator is being inspected according to third-party software (e.g., System Tool Kit (STK)) for simulating the orbital path. The accuracy of the position and / or movement of the stars in the background of the simulated image is being obtained by feeding the simulated image into a plate-solving algorithm and receiving accurate coordinates. A comparison of the resulting image with real satellite images is further inspecting the accuracy.

[0068] The accuracy of the ML model is being further inspected using the training graph and test data set and observing the results.

[0069] This application relates to systems and methods using ML to detect RSOs in the field of stars. The model may be trained on custom-generated simulated images and / or real-world images. Features may be iteratively added to the images, enabling tuning of the dataset with desired features and ensuring their resemblance to real-world images.

[0070] In one embodiment, the structure of the model relates to the Deeptrack 2.0 U-Net model. Adjustments and optimizations were made to the original CNN configuration. Various datasets were used as training inputs, including various epochs, batch sizes, penalties, and loss functions. Once the accuracy results were satisfactory, the model was run on the GPU and the resulting performance data was recorded.

[0071] Given the customizable nature of parameters such as imaging mode and exposure time, an image can be generated similar to any optical sensor.

[0072] Certain advantages of the present system, method, and disclosure include the autonomous nature of detecting RSOs using only satellite or in-space components without relying on ground-based components. Specifically, upon detecting an RSO, subsequent actions may be taken based on the information generated through the RSO detection (e.g., by sensors on the detecting satellite or other satellites). Such subsequent actions may be planned or tasks (e.g., instructions for the sensor to perform specific functionality such as tracking the RSO) may be imposed and may be executed in an autonomous manner (e.g., physically tracking the RSO).

[0073] The present disclosure further relates to a smart space sensor capable of taking autonomous actions to queue another payload sensor (having a high-resolution optical system or hyperspectral sensor) or another satellite, or to continue tracking an RSO. Such capabilities are autonomous and may be performed in or by the smart space sensor without using a ground-based centralized controller.

[0074] Referring now to FIG. 1, a method 100 for detecting an RSO according to one embodiment is shown therein. Method 100 is performed using a computer system or equivalent such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC). The computer system may include one or more computing devices each having a processor and a memory. Method 100 may be stored in the memory as computer-executable instructions and executed by the processor.

[0075] Method 100 may be performed using any suitable type of computing device. Aspects of the method such as inference may be implemented on low-power hardware such as a low-power GPU or FPGA.

[0076] At 102, method 100 includes generating simulated training images using an image simulator.

[0077] At 104, method 100 includes training an RSO detection model using the simulated training images. The RSO detection model is a machine learning model. The RSO detection model may be a neural network-based model.

[0078] At 106, method 100 includes configuring the trained RSO detection model for on-board processing.

[0079] In 108, method 100 includes detecting RSOs in an input image using a trained RSO detection model configured for on-board processing. The trained RSO detection model may be set up or configured to perform inference on low-power hardware such as a low-power GPU or FPGA.

[0080] Next, referring to FIG. 2, a pipeline 10 for generating and using an RSO detector implemented within a CNN according to one embodiment is shown therein. The CNN may be a U-Net. It will be appreciated that the RSO detector may be implemented within a neural network other than a CNN, by a neural network other than a CNN, or together with a neural network other than a CNN, or by an image segmentation model other than a neural network. The RSO detector may be a semantic segmentation model.

[0081] In some cases, some subset of the elements of pipeline 10 may be executed on the ground using one or more computer systems. In a variant, the subset may vary. Such elements executed on the ground may operate without the SWaP problem. In one embodiment, the generation of the completed model / final dataset 17 by the elements of pipeline 10 is executed on the ground (i.e., without the SWaP problem), the completed model / final dataset 17 is sent to the satellite, and the elements that are then executed are adapted to meet the SWaP constraints.

[0082] In one embodiment, the RSO detector is operated on a GPU (see, e.g., GPU 20 below).

[0083] Pipeline 10 is executed by a computer system including a processor and memory or an equivalent implementation of an FPGA.

[0084] Pipeline 10 includes an image simulator 12. The image simulator 12 is stored in memory and executed by a processor. The image simulator 12 is configured to execute an image generation operation 11 to obtain a simulated training data set 14. The data set 14 is stored in memory.

[0085] The image simulator 12 produces images of the same kind as the field of view from an imaging satellite as the simulated training data set 14.

[0086] The simulated training data set 14 includes the presence of RSOs on a photorealistic star background.

[0087] The simulated training data set 14 may be generated according to the method 100 of FIG. 1 at 102.

[0088] The images are rendered with depictions of RSOs and stars to mimic the perspective of a camera on an imaging satellite or a ground-based optical sensor at a given time. The rendered images represent what would physically appear if the images were actually taken at that time and location.

[0089] When the positions of the RSOs and stars are calculated, noise is added.

[0090] In one embodiment, the selection of the satellite may be done for the role of the RSO target (e.g., selection of a GEO satellite). The orbits may be propagated within the simulated training data set 14.

[0091] Within the pipeline 10, once generated, the simulated training data set 14 is curated at 13. The simulator 12 advantageously provides significant flexibility in the curation of the data set 14, allowing for customization for edge cases, different sensor types, different optical systems, etc.

[0092] Pipeline 10 may generate multiple datasets that emphasize various features, where some features have more detailed differences than other features, depending on the reasons, goals, or targets for training the model. For example, a dataset may be provided where all "stars" are significantly dimmer than in the baseline set. In a further example, a dataset may be provided where the streaks of the stars are longer or shorter than in the baseline set. To create the simulated training dataset 14, images from some of the datasets may be taken or combined.

[0093] In 15, the curated dataset 14 is processed to set up the training dataset, test dataset, and validation dataset.

[0094] Since the curated dataset 14 closely resembles data taken by real satellites, pipeline 10 includes an RSO detector machine learning model training operation in 16. The model is trained using the curated dataset 14. The training of the model 16 may be performed according to the method 100 of FIG. 1 in 104.

[0095] The model training in 16 generates a completed model as the output in 17, and such a model is provided as the input to the inspection operation 18.

[0096] If the inspection operation 18 fails, the retraining operation 21 is started. The retraining is executed using the model training operation 16.

[0097] The work cycle 10 includes the inspection of the model in 18 to produce an inspected model if the inspection is successful. If the inspection of the model in 18 fails, the model 16 is retrained.

[0098] The inspection operation 18 generates an inspected model. The inspected model is performance - tested 19 by operating the graphics processing unit (GPU) 20 based on the trained model 16 so as to be inspected at 18.

[0099] Next, referring to FIG. 3, the image simulator 12 of FIG. 2 is shown therein in more detail according to one embodiment.

[0100] The image simulator 12 includes a foreground generator 122 for generating the foreground of the simulated training data set 14.

[0101] The image simulator 12 further includes a background generator 124 for generating the background of the simulated training data set 14.

[0102] The image simulator also includes an image generator 126.

[0103] At 202, the foreground generator 122 calculates the coordinates of the image satellite at a given time. The coordinates of the image satellite are calculated using a propagation algorithm (e.g., Simplified General Perturbations 4 (SGP4)).

[0104] At 204, the foreground generator 122 determines the region of interest for which the field of view (FOV) of the image satellite at a given time is given. For a particular FOV and region of interest, when the information on where the satellite is pointing at a given point in time is known, a database of star positions may be browsed to find that area. When the coordinates of the image satellite are known, the FOV is calculated using trigonometry.

[0105] At 206, the foreground generator 122 calculates the coordinates of all other RSOs and saves those RSOs inside the region of interest.

[0106] At 208, the background generator 124 queries a database for the coordinates of all stars that fall within the FOV. For example, the database may be the Tycho2 database, which is a public database that stores constellation coordinates, brightness, and other stellar information.

[0107] At 210, the background generator 124 adds noise to the simulated training dataset 14. For example, various types of noise may be added by applying a Gaussian noise function through Python. Other exemplary noise includes star streaks. Star streaks may be added by drawing a streak on either side of the current position of the star.

[0108] At 212, the background generator 124 converts the RSO and star coordinates to pixel coordinates within the simulated training dataset 14. The conversion may be performed through a mapping function.

[0109] At 214, the image generator 126 selects an imaging mode from among stellar tracking, target rate tracking, and custom rate.

[0110] At 216, the image generator 126 sets the exposure time of the image to be rendered.

[0111] At 218, the image generator 126 renders the image. The image may be rendered through Image.render, i.e., a built-in function in Python.

[0112] Next, referring to FIG. 4, a computer system 300 according to one embodiment is shown therein.

[0113] The computer system 300 includes a processor 302 for executing software models and modules.

[0114] The computer system 300 further includes a memory for storing data, including the data output from the processor 302 and the simulated image 320.

[0115] The processor 302 is configured to execute an image simulator 305. The image simulator 305 may be the image simulator 12 of FIG. 1.

[0116] The image simulator 305 includes a foreground generator module 306, a background generator module 308, and an image generator module 310.

[0117] The foreground generator 306 is configured to generate the foreground of the simulated training data set.

[0118] The foreground generator 306 calculates the positions of all initialized satellites and depicts the satellites at their exact positions at a given time. To do so, the image satellites are initialized. Using visualization software (e.g., STK) configured to simulate orbital paths, a list of access windows is generated. These access windows are the times when the foreground satellites are within the FOV of the image satellites.

[0119] Since the access windows are finite (STK only generates timestamps for the time an object is within the FOV), the timestamps are slightly randomized.

[0120] Such randomization advantageously ensures that the satellites are not always centered in the images produced by the foreground generator 306.

[0121] Such randomization further advantageously expands the data sets for training, validation, and testing.

[0122] The foreground generator 306 takes timestamps and calculates the right ascension (RA) and declination (DEC) of all initialized satellites at a given time.

[0123] The RA and DEC of the imaging satellite are set to be the center of the generated image. The RA and DEC of all other satellites are also calculated at a given time.

[0124] The real-world RA coordinates and DEC coordinates are converted into pixel coordinates used to render the image.

[0125] These data points are stored in the foreground table 316, which should be combined with the background table 318 filled by the background generator 308. The foreground table 316 and the background table 318 are stored in the memory 304.

[0126] It is also important to note that the size of the simulated image is specified to match the size of the image of the satellite camera on which the model trained by the simulated image is deployed. Pixel coordinate entries are calculated for all satellites and then saved to be rendered in the final image if they are inside the FOV.

[0127] The foreground generator 306 uses accurate orbital information to project all satellites to their exact positions within the frame of the simulated image 320.

[0128] Using an analytical or numerical propagator or propagation method, the classical orbital elements of each satellite are utilized to determine its coordinates given a time stamp. In one embodiment, the propagation method is SPG4. Any suitable analytical or numerical propagator or propagation method may be used.

[0129] The image simulator 305, and in particular the foreground generator 306, propagates the imaging satellite and uses the configurable FOV angle attribute of the imaging satellite to determine the aerial area that the satellite will be able to "see" at a given time.

[0130] All other initialized satellites are propagated to determine their respective RA and DEC coordinates.

[0131] If the foreground satellite enters the FOV of the imaging satellite, the coordinates of the foreground satellite are input into the rendered image.

[0132] To introduce an illusion of varying brightness at the RSO and stars due to distance, relative light emission levels, and reflection and refraction characteristics, and to generate realistic brightness variations across the overall simulated image 320 by incorporating pseudo-random elements, a configurable scale of brightness is applied by foreground generator 306. The brightness parameters may be extracted from a star catalog such as Tycho2 under the "brightness" scale of stars or celestial objects in the green portion of the visible light spectrum, i.e., Gmag. Random brightness may also be used. When random brightness is used, it may be convenient not to provide many bright saturated stars that can affect astrometry.

[0133] The size of the RSO drawn on the simulated image 320 is randomized between 3 and 5 pixels so as to best represent the reference data.

[0134] Additionally, the brightness is randomized. In one embodiment, the brightness is randomized between visual magnitude 9 and visual magnitude 14. Visual magnitude is a logarithmic astrometric value that refers to the apparent brightness of a star, not its absolute brightness. For example, a star may be extremely bright compared to the sun, but may appear faint if it is far away.

[0135] Referring again to the image simulator 305, the background generator 308 is configured to draw a field of stars in the background of the simulated training data set, i.e., behind the RSO.

[0136] The background generator 308 queries the star database for all stars whose current RA and DEC fall within the FOV of the imaged satellite. The star database is stored in the memory 304. The FOV of the imaged satellite is specified in their configuration files and is, for example, 1.4°.

[0137] The background generator 308 further converts the RA and DEC of the stars falling within the FOV radius to pixel coordinates.

[0138] The background generator 308 further stores the pixel coordinates in the background table 318.

[0139] The background generator 308 generates the star background of the simulated image 320.

[0140] Utilizing the background table 318 containing the positions of the stars in the sky, the background portion of the simulated image 320 is filled with exact stars that would fall within the FOV of the imaged satellite given their coordinates at the specified time.

[0141] All the coordinates of the stars falling within the field of view of the imaged satellite payload (i.e., the camera) are collected and stored in the background table 318.

[0142] Using the same process appropriate for the foreground satellite coordinates, the constellation coordinates are converted from the RA and DEC positions to pixel coordinates in the simulated image 320.

[0143] Once the background portion of the simulated image 320 is rendered, the accuracy of the background portion of the simulated image 320 may be verified using a plate solving algorithm, which was able to successfully determine the rendered portion of the night sky.

[0144] In one embodiment, the database is the Tycho2 database, which contains the positions of 2.5 million of the brightest stars in the sky.

[0145] The image simulator 305 creates a simulated satellite image. The image simulator 305 produces an image of the same kind as the field of view from an imaging satellite and completes it with the presence of known RSOs on a photo-realistic star background. In one embodiment, the image simulator 305 is written in Python.

[0146] In one embodiment, the selection of a satellite (e.g., a GEO satellite) serves as an RSO target. The image simulator 305 advantageously allows for customization for any one or more of edge cases, different sensor types, different imaging satellites, etc. to provide sufficient flexibility during the curation of the simulated training dataset 14 of FIG. 2 for training the AI model.

[0147] Advantageously, the image simulator 305 can generate an image of a customizable satellite (e.g., a GEO satellite) that will be within the field of view (FOV) of an imaging satellite (e.g., in LEO) at any given time. The image simulator 305 may generate an image of any suitable satellite that will be within the FOV of an imaging satellite in a lower orbit. All orbit propagations are accurate. At the exact location in the simulated image 320 output by the processor 302, a satellite with a higher orbit is depicted over the image.

[0148] The image simulator 305 further queries a star database (e.g., the Tycho2 database) to fill all the stars that fall within the FOV of LEO at a given time.

[0149] To make these images as realistic as possible, several customizable noise and distortion methods are implemented, including streaking the stars to simulate an open aperture during the high speed of the orbit.

[0150] To make the simulated image 320 as closely similar as possible to the actual satellite image, the image simulator 305 provides several configurable features for customizing the simulated image 320.

[0151] The image simulator 305 may apply Gaussian noise to the RSO and stars. The resulting effect is a blurring of the edges of both features similar to what might be seen in a real image.

[0152] The image simulator 305 may apply diffraction spikes, such as those seen in a reflector or Cassegrain telescope system.

[0153] The image simulator 305 may further apply background noise in the black space of the background of the simulated image 320.

[0154] The background noise simulates camera defects, light in the sky, as well as hot pixels and dead pixels. The background noise may appear as a faint fuzzy gray in the background of the image. The user of the computer system 300 may optionally engage or disengage the background noise function.

[0155] The image simulator 305 may impart a visual effect into the simulated image 320 to simulate the effect of an open aperture in an orbiting camera.

[0156] Such a visual effect includes streaking the background stars. Streaking is achieved by rotating the background table 318 forward and backward over time and overlaying the resulting image into the simulated image 320. The user of the image simulator 305 may optionally engage or disengage the streaking function.

[0157] In reality, if the RSO is directly above a background star from the perspective of the camera of the imaging satellite, the RSO is not visible. Similarly, if a foreground star is directly above a background star from the perspective of the camera, the background star is not visible.

[0158] Therefore, a further visual effect available in the image simulator 305 is the removal of any such overlaps in the simulated image 320. Such removal of overlaps is performed by checking the pixel coordinates of the stars in the background table 318 against the pixel coordinates of the RSO and the stars in the foreground table 316.

[0159] In the case of an overlap between a foreground star and a background star (furthermore, when the star is close enough from the perspective of the camera), the overlapping background star is not drawn in the simulated image 320.

[0160] In the case of an overlap between a foreground RSO and a background star, the overlapping foreground RSO is not drawn in the simulated image 320.

[0161] When the streaking of stars is enabled, all the pixels where the star streaks will be drawn are checked, and if there is an overlap anywhere in the streaks, the entire star is not drawn.

[0162] The user of the computer system 300 may be able to selectively engage or release the removal of the overlap function.

[0163] An interesting feature implemented in the background generator is the removal of background stars that would be drawn over the RSO. In a real-world scenario, an RSO precisely aligned with a star would be undetectable. In an operational environment, a series of multiple frames may be analyzed, enabling the RSO to move and become visible again. This feature facilitated the future training of the model without compromising the likely operational flow. Alternatively, stars or streaks of stars that would otherwise obscure the RSO need not be drawn in a way that enables the detection of the RSO.

[0164] The image simulator 305 may generate an image having a random number of RSOs at random locations on a random background. A user of the computer system 300 may selectively engage or disengage the above-described random generation feature.

[0165] Referring again to the image simulator 305, the image generator 310 is configured to generate a simulated image 320.

[0166] The image generator 310 generates the simulated image 320 by appending the background table 318 to the foreground table 316 and depicting the result.

[0167] If additional visual effects are to be incorporated into the simulated image 320, the image generator 310 applies those visual effects.

[0168] To verify the validity of orbit calculations based on the simulated image 320, the calculations based on them were compared to a generated list of coordinates from STK.

[0169] To verify the legitimacy of the positions of the background stars in the simulated image 320, the fully rendered simulated image 320 was fed into the plate-solving algorithm. The plate-solving algorithm returned the calculated coordinates of the image satellites corresponding to the coordinates of the center of the simulated image 320. The plate-solving algorithm is used to determine which part of the night sky is being viewed by triangulating the detected stars and recognizing patterns based on a database. Astronomers feed images from telescopes into these algorithms to determine where in the night sky they are observing. The above demonstrates the accuracy of the simulated image 320 by proving that the positioning of the stars in the simulated image 320 represents the actual night sky.

[0170] To verify the positions of all RSOs in the simulated image 320 generated by the system 300, visualization software (e.g., STK) for simulating orbital paths was utilized.

[0171] In STK, the simulation was constructed from all the satellites propagated by the foreground generator 306, including all the satellites used in the simulated image 320.

[0172] An access window representing the time when the satellite entered the FOV was calculated. The relevant timestamps were used to generate the simulated image 320.

[0173] To verify the accuracy of the Python propagation function used in the image simulator 305 for calculating RSO coordinates, the STK propagation function was further used.

[0174] Various noise attributes were added to simulate images on orbit or not on Earth.

[0175] First, the image simulator 305 generated the RSO and the stars as two-dimensional circles with high contrast between the object itself and the dark background space.

[0176] The resulting image can be likened to a black picture with white dots.

[0177] Since this does not represent the source image, time distortion was added to simulate the actual camera shutter speed and introduce noise similar to the conditions faced by low Earth orbit (LEO) satellites.

[0178] This addition ensured that the stars in the background appeared as streaks rather than symmetric circles.

[0179] The added noise profile was multi-stage.

[0180] The foreground has Gaussian noise added to "soften" the edges of the RSO.

[0181] The background has Gaussian noise added to simulate background cosmic radiation and "soften" the edges of the existing star streaks. In addition, functionality was provided to add hot pixels and dead pixels representing an undesirable but common incidence rate in image capture devices.

[0182] Hot pixels are characterized by always being on, and dead pixels are characterized by always being off.

[0183] All of this noise, time distortion, and pixel options are configurable, except for the noise added to the RSO.

[0184] The RSO may appear to have "soft" edges regardless of the imaging device, but the remaining noise may be attenuated by newer and more sophisticated noise attenuation techniques and radiation enhancement.

[0185] Next, referring to FIG. 5, a flowchart of a method 400 for generating a simulated image to be used as a training image for training an RSO detection model according to one embodiment is shown therein.

[0186] Method 400 may be suitable for training a U-Net or other CNN-based model. Method 400 may be executed by the computer system 300 of FIG. 4. Method 400 may generate the simulated image 320 of FIG. 4. Method 400 may generate a simulated image according to the method 100 of FIG. 1 at 102.

[0187] At 402, method 400 includes generating an image (e.g., the dataset 14 of the pipeline 10 of FIG. 2) to be used for training the model.

[0188] At 404, method 400 includes depicting any satellite flying higher than the imaging satellite over the generated image at the exact location. This may include depicting images of satellites in various orbital regimes that would be within the field of view (FOV) of a low Earth orbit (LEO) imaging satellite at any given time.

[0189] At 406, method 400 includes querying a star database to fill all the stars that fall within the FOV of the LEO satellite in the generated image.

[0190] At 408, method 400 includes applying customizable noise and distortion methods to the generated image. Such noise and distortion methods may advantageously make the generated image as realistic as possible.

[0191] At 410, method 400 includes applying streaking of stars to simulate an open aperture during the high speed of the orbit of the LEO imaging satellite.

[0192] Next, referring to FIG. 6, a schematic diagram of a modified U-Net CNN segmentation model 600 for detecting RSO according to one embodiment is shown therein.

[0193] The U-Net model 600 is trained to detect RSO.

[0194] The U-Net 600 may use the simulated image 320 generated by the image simulator 305 of FIG. 4 to train the model to detect RSO.

[0195] The U-Net 600 includes a plurality of processing nodes 612 configured in a plurality of layers corresponding to a sigmoid function and including an input layer 602, at least one hidden layer, and an output layer having at least one output node.

[0196] The U-Net 600 includes an input layer 602 for receiving an input to the U-Net 600. The input may be one or more simulated images 320.

[0197] The U-Net 600 further includes hidden layers 604-1, 604-2, 604-3, 604-4, 604-5 for processing the output of the input layer 602. It will be appreciated that any number of hidden layers may be provided.

[0198] The hidden layers 604-1, 604-2, 604-3, 604-4, 604-5 are hidden in that the details of the processing are not known or are not shown outside the U-Net 600.

[0199] As shown in FIG. 6, any number of hidden layers 604-1, 604-2, 604-3, 604-4, 604-5 may be present.

[0200] The U-Net 600 further includes an output layer 610 for receiving the output of the hidden layers 604-1, 604-2, 604-3, 604-4, 604-5 and generating an output. The output may be an output segmentation map.

[0201] In one embodiment, the output of the U-Net 600 is a trained model. The trained model may be an arrangement or instantiation of the U-Net 600. The trained model may be provided to the U-Net 600 as an input, or alternatively, a set of parameters or other values used to define or modify the U-Net 600 so as to affect or act on the trained model or otherwise.

[0202] The U-Net 600 is a convolutional neural network (CNN) whose architecture advantageously provides fast and precise segmentation of input images (e.g., simulated image 320) through modification and expansion.

[0203] The U-Net 600 with the modified and expanded architecture advantageously trains the model using smaller inputs (e.g., a smaller number of simulated images 320 provided as training images).

[0204] When trained, the model results in a more precise segmentation of the input (e.g., of the simulated image 320).

[0205] When implemented on a modern graphics processing unit (GPU), the segmentation of a 512×512 image through the modified U-Net model 600 may take less than one second.

[0206] The modified U-Net 600 supplements the conventional CNN with successive layers (e.g., a larger number of hidden layers 604).

[0207] The pooling operation of the conventional CNN is replaced by an upsampling operator.

[0208] Each successive hidden layer 604-1, 604-2, 604-3, 604-4, 604-5 learns to assemble a precise output based on the output of the previous layer. Thus, a larger number of layers advantageously increases the resolution of the trained model.

[0209] One of the above-mentioned modifications applied to the architecture to generate U-Net600 is the large number of feature channels in the upsampling operator to enable U-Net600 to propagate context information to higher-resolution layers.

[0210] Thus, the expansion path of U-Net600 is more or less symmetric with the contracting part of U-Net600.

[0211] As a tiling strategy, U-Net600 uses only the valid part of each convolution that does not have any fully connected layers.

[0212] To predict the pixels in the boundary region of the simulated image 320, the lost context is extrapolated by mirroring the simulated image 320 as input.

[0213] The above tiling strategy advantageously assists U-Net600 when processing large images, and without using that strategy, their resolution may be disadvantageously limited by the GPU memory.

[0214] Thus, the above tiling strategy advantageously enables the implementation of U-Net600 on low-power GPUs.

[0215] U-Net600 includes a contracting path and an expansion path (not shown).

[0216] The contracting path and the expansion path together result in a U-shaped architecture.

[0217] The contraction path includes repeated applications of convolution, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction path, spatial information is reduced while feature information is increased.

[0218] The expansion path combines feature information and spatial information through a sequence of up-convolutions and concatenations using high-resolution features from the contraction path.

[0219] In one embodiment, the architecture of the RSO detector convolutional neural network (CNN) 600 was constructed using TensorFlow 2.6.2 and Keras 2.6.0 layers with Python 3.6. The basis of the model is from the DeepTrack 2.0 U-Net model. The U-Net 600 or a model trained against it has proven to successfully identify small clusters of pixels in an image, which is important for detecting RSOs. Conventional U-Net models may have approximately 31,000,000 trainable parameters, while the modified U-Net 600 has, in one embodiment, 46,898, i.e., a 99.8% reduction. Advantageously, the modified U-Net 600 incorporates a significant reduction in nodes. Such a reduction not only speeds up the training process but also further improves the on-board performance of the model. Additionally, the reduction allows the training to direct more attention to the remaining weights. Convolutional layers and transposed convolutional layers are used to determine the location of the RSO in the input image provided to the input layer 602. The model is compiled using the "Adam" optimizer, the main metric is "accuracy", and the loss function is tensorflow_addons.losses.SigmoidFocalCrossEntropy().

[0220] The modified U-Net600 includes a SigmoidFocalCrossEntropy loss function. Such a focal loss function is better adapted to imbalanced datasets such as images of the universe containing RSOs. Even when there are 8 RSOs in an image, more than 99% of the pixels were found to not be RSOs, but rather star streaks or background. Since the focal loss function takes the imbalance into account when training the model, such a large imbalance is partially compensated for by the focal loss function, leading to better weight values during training.

[0221] Tests and Experiments One embodiment of the RSO detector model of the present disclosure was trained using 320 simulated images. The final model looked at 3484 images, 2800 of which were used during training and the rest were used for per-epoch validation. A batch size of 32 was used during the 25-epoch run. The model was then tested against other datasets. All images were 256×256 grayscale PNG files in order to save space and training time without sacrificing the information captured in the images. The length and direction of star streaks, background noise, and the number of RSOs varied between images to create various datasets. The above dataset size balanced the need for sufficient training data while avoiding overfitting. The selection of batch size and number of epochs was also the result of iterative investigation. Below 20 epochs, the model consistently failed to predict. When more than 25 epochs were provided, at best only marginal improvement was shown or it led to overfitting.

[0222] Overfitting and underfitting are both problems related to models in machine learning. A set of images was devised to train the model such that each has a different number of details that provide sufficiently unique or distinct variables so that the training does not start focusing on details. Non-exhaustive examples of overfitting encountered and mitigated during this project include the following, namely, streaks that are always in one orientation and of the same length and thickness, all RSOs being of a certain brightness or always brighter than the stars, and the fact that a fixed amount of RSOs is present in the image or that the presence of RSOs in the image is guaranteed.

[0223] The above problem was mitigated by introducing new training data with newly randomized details to prevent all images from having similar features.

[0224] Traditionally, F1 and intersection over union (“IoU”) scores have been used to calculate the accuracy of a model. Due to the unbalanced nature of the dataset, the way being influenced by the quantity of object pixels versus non-object pixels did not yield representative results of the prediction accuracy. When small-sized RSOs (in terms of the number of pixels representing RSOs) are given in the simulated image 320, the distance between the center of the true mask RSO and the center of the predicted RSO was considered to yield a more representative metric. In machine learning, a true mask refers to a boolean mask indicating ground truth values for a specific task. In image segmentation, a true mask is a binary image where each pixel is labeled as either belonging to the object or not. In this case, the true mask is an image representing whether each pixel is an RSO pixel or not. This mask serves as the reference or ground truth against which the model's predictions are evaluated.

[0225] The centers of each pixel cluster in any of the images were found via the centroid method and recorded in a list. Two lists were compared to see whether the RSO was predicted to fail or whether the prediction had a current false positive. If the centers of these pixel clusters were very close to the known RSO, the prediction was considered a success. If the centers were not close, the model was considered to have lost the target. In the simulated images, the RSO typically has a width of 3 pixels, so if the centers were within 5 pixels of each other, the prediction was considered a success. The centers of the prediction and the true mask were found to deviate by an average of only half a pixel, demonstrating that the location of the prediction was very accurate. Since the distance metric is not based on the overall percentage of detected object pixels, but rather on the number of RSOs in the image, the distance metric represents a more meaningful metric for measuring the accuracy of the model. Such experiments were later repeated using real images.

[0226] The most performant of several models that were generated and trained were selected for subsequent evaluation of accuracy and error.

[0227] An analysis was performed on the prediction certainty of the model. The model takes in grayscale images as input. The model normalizes the data in the input grayscale image from 0 to 1, i.e., assigns a boolean value of 0 or 1, representing black or white respectively, to each pixel of the image. The model interprets the gray color to determine whether each pixel is part of an RSO. The model outputs a similarly normalized prediction. This threshold represents the certainty of the model that the pixel is an RSO or part of an RSO. The output of the model is a boolean matrix (a matrix containing numbers between 0 and 1 corresponding to each pixel in the image). This number (0 - 1) represents the certainty of the model that the pixel is an RSO. For example, if 0.7 is assigned to a certain pixel, the model is 70% certain that the pixel is an RSO pixel.

[0228] Various thresholds were tested to identify which produced the best threshold, starting with pixels above 0.50 and proceeding down to 0.10. Such a threshold means how confident the model had to be that the pixel in question was an RSO for the output to be counted as an RSO. The optimal threshold encountered, which means the model should have been 50% certain that the pixel was an RSO for the pixel to be counted in the output, was 0.5. If the model considered there was a 40% probability that the pixel was an RSO, the model rounded that probability down to 0.

[0229] Figure 7 shows a graph of the analysis of the effect of the threshold on the accuracy of the trained model across multiple datasets, including star streaks of random length and width and low to medium levels of noise. During training, some datasets were used to include filtered edges to prevent star streaks from appearing on the edges of the test images, as the model would advantageously target and focus on these cases. Each dataset is a collection of simulated images and their associated ground truth masks for verification.

[0230] Multiple datasets with different features were combined to create a large dataset with many different types of images. The features included in the large dataset were as follows. · Star streaks of random length and width, · Low to medium levels of noise, · Filtered edges, as well as · Normal edges.

[0231] As the threshold increased, the false positives tended towards a 0% occurrence rate, but the disappearing RSOs began to increase. As the threshold decreased, the disappearing RSOs were reduced towards 0, but false positives were reported more frequently.

[0232] The threshold of 0.25 showed the optimal balance between false positives and disappearance detections that aligned with the overall maximum of successful predictions. This threshold was calculated specifically for the model selected for analysis. If this experiment is repeated using different models or for different data, this threshold will be retuned through a similar analysis.

[0233] When using a model trained with U-Net for RSO detection in an operational system context, both false positives and disappearing RSOs may be reduced through the use of tracklet (or time series) data as well as complementary on-board data sources. If a given data source (e.g., a sensor serving as a complementary on-board data source) is a primary (i.e., direct) input to the trained model, the model's threshold is tuned to not miss RSOs and a verification subroutine is used to filter out false positives. For this test and experiment, the threshold was selected to optimize the intersection of both the minimum number of disappearing RSOs and false positives.

[0234] Several metrics were used to quantify the results of the model. "Average distance" refers to the amount of pixels between the center of the ground truth mask RSO and the center of the predicted RSO. The average distance measures the accuracy of the location positioning of the model's predictions. The metrics described in Table 1-1 summarize the results seen from the validation data set, which includes lighter RSOs, limited background noise, and no prominent streaks of stars (sometimes found in the input images). The results from the validation against a clean data set such as the above validation data set are extremely good.

[0235] [Table 1]

[0236] This model is being further tested to understand its strength and limitations. This was achieved by validating against a dataset with more features and added noise than those used during training. As expected, there was a slight reduction in accuracy, but the model remained accurate in its predictions. Table 1-2 (Table 2) shows the results from a dataset with much thicker star streaks and much higher levels of background noise than the set used during training.

[0237]

Table 2

[0238] From the results of Table 1-2 (Table 2), the trained model shows an accuracy rate of 83.8%, where each prediction has an average distance of 0.5 pixels from the True Mask center.

[0239] Small detection errors are introduced by a number of factors. Sometimes, the RSO is extremely faint and the model fails to detect it. In other cases, the RSO was near a star streak and was misinterpreted as part of the star structure. Most false positive detections appear to occur at the endpoints of the star streaks. It is possible that the model misinterpreted the rounded edges of the star streaks as RSOs. The overall error rate is low even on a noisy dataset, and it is possible to improve the accuracy by further training using additional datasets or by using time series data.

[0240] The average distance between the predicted location and the True Mask location is extremely short, usually deviating by only half a pixel from the True Mask location. This result demonstrates that the RA and DEC coordinates generated from the predictions are extremely accurate and can be used during orbit calculation and orbit propagation. Using additional datasets for training and testing may further reduce the error margin in the coordinates returned from the model's predictions.

[0241] In one embodiment, the input to the trained U-Net model includes an image having a single channel and dimensions of 256×256 pixels. The image is normalized along the grayscale. Such normalization may be achieved by dividing the values of the input image (i.e., pixels) by a factor of 255 (equivalent to 0xFF in hexadecimal representation).

[0242] The output in the above embodiment (e.g., referring to FIG. 10) also includes an image having a single channel and dimensions of 256×256 pixels. Advantageously, the output image displays only the RSOs therein if any RSOs were present in the original image. Thus, the true mask image used during model training for verification is generated to include only "on" or "off" pixels. A pixel is "on" only if an RSO is present and "off" otherwise. There is no noise or any other artifacts in the generated true mask image.

[0243] Referring now to FIG. 8, a computer system 700 for implementing a modified U-Net CNN 600 according to one embodiment is shown therein. In FIG. 8, like numerals indicate like references with respect to FIG. 4.

[0244] The low-power processing unit 702 advantageously performs the functions of a processor in a low-power context, for example, during the operation of a robotic device after launch in space where excessive hardware is not available for computing. The low-power processing unit 702 includes a low-power GPU for performing the above functionality within the low-power processing unit 702.

[0245] Memory 704 includes a model 732 for detecting RSO. Model 732 is as described with respect to FIGS. 4-8. Model 732 may be generated by a U-Net such as U-Net 600 of FIG. 6. Model 732 is generated by training a modified U-Net 600 that outputs model 732. Thus, model 732 may be regarded as already trained.

[0246] Model 732 is loaded and executed on low-power processing unit 702. Model 732 receives an input 735 (e.g., a further image) in order to detect an RSO in the input 735 or make a prediction regarding the RSO therein.

[0247] In one embodiment, model 732 was executed on an Xavier Jetson AGX GPU. Model 732 was created and its variables were frozen. Model 732 was converted to the ONNX format for running inference on low-power GPU 730. The newly generated ONNX file 732 was loaded onto low-power processing unit 702. An execution file was set up on computer system 700 to capture input 735 (e.g., an input image) and perform predictions using the trained model 732.

[0248] Experiments were conducted regarding the impact of various power modes provided by Xavier Jetson AGX. Xavier Jetson AGX provides various power budgets, CPU / GPU frequencies, number of cores, and other options. There are a total of eight modes, as outlined in Table 1-3 (Table 3).

[0249]

Table 3

[0250] In Table 1-3 (Table 3), DLA represents a deep learning accelerator and PVA represents a programmable vision accelerator.

[0251] Each of these modes was tested, and the trained model 732 performed inferences on a set of 100 test images as input 735. The inference speed refers to the time required for the model 732 to process the input 735 and make predictions.

[0252] Next, referring to FIG. 11, a chart showing the inference speed versus the GPU power mode according to one embodiment and displaying findings based on statistical values of various power modes is shown therein. Using 32 floating-point data input in the reference, modes 0 and 6 showed the best results.

[0253] Referring back to FIG. 10, from the general trends shown above, it can be seen that the CPU frequency plays a greater role in the inference speed of the trained model 732 rather than the power allocated to the low-power processing unit 702. It is concluded that power mode 7 is most closely similar to the space-qualified low-power processing unit 702 and should be available for predictions to be made in approximately 23 ms with expansion.

[0254] Next, referring to FIG. 12, a chart showing the CPU maximum frequency versus the inference speed according to one embodiment is shown therein. FIG. 12 shows the expected trend in that the time required for inference decreases as the frequency of the low-power GPU 730 increases. FIG. 14 further shows that at lower frequencies, FP32 (single-precision floating-point format) is faster than FP16 (half-precision floating-point format). However, as the frequency of the low-power GPU 730 increases, that difference becomes less important. As defined above in Table 1-3 (Table 3), multiple data points at 1200 MHz correspond to various power modes operating at that frequency.

[0255] Referring again to FIG. 10, when a space-qualified final low-power GPU 730 is selected for a particular flight, these analyzes may be performed again to optimize the exact hardware identified for the mission system.

[0256] System 700 may advantageously be suitable for providing a closed-loop decision support solution for on-orbit situational awareness and autonomous flight control.

[0257] System 700 may advantageously enable an image simulator (such as image simulator 305 of FIG. 4) to model time-series data including a wider range of optical payloads.

[0258] System 700 may advantageously enable the use of on-orbit SSA payload mission data, including a wide temporal archive of image orientations, features, and errors. In conjunction with images using a wide FOV sensor, such enablement may advantageously result in a significant expansion of capabilities over the above disclosure.

[0259] The dataset to be used in the prototype of the above is modeled after an image with a limited FOV.

[0260] The presence of far more objects in the frame and the greater computational workload can further enable the training of detection algorithms for images from a hemispherical lens.

[0261] Image simulator 305 is configured to adjust the luminance composition to emulate the RSO using various lighting conditions and angles of incidence.

[0262] In one embodiment, a more configurable type of noise is added, including simulating non-responsive pixels or dead pixels whose output varies and does not provide consistent readings.

[0263] In one embodiment, the image simulator 305 simulates an image 320 that has a broad temporal dataset to enable training of a model 732 that uses a complete catalog of changing actual view grades in the RSO, as well as a larger set of various image features.

[0264] The capabilities of the trained model 732, trained according to the above, include the detection and tracking of low Earth orbit (“LEO”) satellites.

[0265] The system 700 may be configured to use batch or tracklet images (images captured over several seconds). Such batch or tracklet images may advantageously represent more of the operational payload data for an SSA mission. Those image segments (not shown) improve the detection accuracy in cases where an RSO may overlap with background stars for a single image frame. The preliminary results of the above experiments were adjusted to take this into account, as it is assumed that the time series input data will mitigate this issue.

[0266] Other suitable devices for implementing the computer system 700 may include the Innoflight CFC-500.

[0267] Another suitable device is the CFC-500, which supports the Compute Unified Device Architecture (CUDA) runtime environment. Such an environment advantageously allows for the transfer of trained models 732 with minimal complexity. Since system 700 has an architecture and runtime environment similar to the hardware that implements the modified U-Net 600 on which the trained model 732 was generated, the trained model 732 advantageously executes in the same manner on system 700. The low power requirements of such devices represent an advantage in constrained systems common to satellites in orbit. The CFC-500 further includes two Nvidia Tegra K1 chips, which may reduce the inference time and may be suitable for processing a wider range of systems.

[0268] In one embodiment, the above disclosure further includes distinguishing RA coordinates and DEC coordinates from optical images.

[0269] When range information remains important for orbit determination and collision risk assessment by extension, the present disclosure expressly contemplates the introduction of multi-modal input types, such as radar, LiDAR, and even stereo images, to the SSA system.

[0270] Using multi-modal input, the system enables the spacecraft to provide its own situational awareness, which is important for a closed-loop decision support system.

[0271] In one embodiment, the SSA OBP enables an end-to-end prototype system for on-board detection of RSOs in optical image data.

[0272] The systems, methods, and devices of the present disclosure are developed using ML algorithms created using simulated images of RSOs in the background of stellar filaments in space.

[0273] Using thousands of simulated images, a modified U-Net CNN model was successfully trained with a 99.8% reduction in weights from the standard model. The various details of the simulated images demonstrated that the model could successfully predict among a number of lighting conditions.

[0274] The accuracy of the model was evaluated by measuring the distance between the center of the predicted RSO and the center of the true mask RSO. Examination of the predicted images showed that the location of the predicted RSO was highly accurate, often off-center by at most half a pixel with an accuracy of 83%.

[0275] When executed on a GPU, this model was able to perform inference at a speed of approximately 23 ms at a power budget of around 15 W and a maximum frequency of 2000 MHz.

[0276] The model can be retrained for different input image sources.

[0277] The above demonstrates the utility in the SSA region and enables various applications from initial collision detection systems to autonomous spacecraft steering decisions.

[0278] Next, referring to FIG. 9, a method 800 for implementing a trained RSO detection model on a GPU according to one embodiment is shown therein.

[0279] The trained model may be the trained model 732 of FIG. 8. The GPU may be the low-power GPU 730 of FIG. 8. The GPU may be, for example, a Xavier Jetson AGX GPU or the like. The trained model may be trained according to the simulated image 320 of FIG. 4. The method 800 may implement the method 100 of FIG. 1 at 106.

[0280] At 802, the method 800 includes freezing the trained model.

[0281] When the model is trained, the model maintains variables (e.g., weights) within its layers. As a result, if training continues, the variables may be updated to yield greater accuracy, i.e., the model learns during training. Freezing the graph makes the variables constants, meaning that the model at that point no longer undergoes further training or modification. The resulting trained model may be read by a program, especially as a smaller file.

[0282] At 804, method 800 includes transferring the frozen model to a GPU.

[0283] In one embodiment, the model, and the files supporting the model, are created using Python 3.6 TensorFlow 2.x in h5 and pb formats. Freezing the model at 802 allows for conversion to the UFF format.

[0284] At 806, method 800 includes converting the transferred frozen model to the Open Neural Network Exchange (「ONNX」) format.

[0285] The Unified File Format (「UFF」), which has been common in the art to date, is an older format that has lost developer support. Moreover, UFF presents single-channel input / output where the 「reshape」 operation interferes with the model's predictions. Advantageously, converting the transferred frozen model to ONNX provides better support.

[0286] The converted ONNX model is uploaded to the GPU.

[0287] At 808, method 800 further includes creating an engine file, which is a pre-compiled version of a model (such as model 732) optimized for a specific GPU architecture. In one embodiment, the engine file is created using Nvidia's TensorRT library. The engine file and the ONNX file are different representations of the model. Advantageously, the engine file may convert the ONNX file into various formats that the GPU can understand.

[0288] The engine file executes the frozen model transferred in the ONNX format. Advantageously, the engine file may make modifications to optimize the model for execution on the GPU.

[0289] At 810, method 800 includes operating the engine.

[0290] To enable a determination of which type of inference engine is optimal for a given model, the performance of the inference may be recorded. Inference refers to the action of making a prediction. The model operates on a piece of hardware called an inference engine. Multiple inference engines are provided on the Xavier Jetson AGX GPU.

[0291] At 812, method 800 includes inspecting a dataset produced by a trained model executed on the engine at 810.

[0292] In one embodiment, when a model is generated, the model is retained in a hierarchical data format (HDF), such as the "h5" (HDF5) file format. HDF5 is used to store large amounts of data in the form of multi-dimensional arrays. The dataset may be made from images generated by an image simulator. The number of images and the specifications of their content may vary among them based on the configuration options at the time of generation, but there are some areas that are the same among all datasets. These are that for every simulated image, a corresponding mask image containing only RSO pixels is also generated, that each simulated image contains star streaks, and that each simulated image and the corresponding mask image are square.

[0293] Additional non-exhaustive details that are variable among the datasets, including the length and thickness of the star streaks, the level of noise present in the simulated images, and the amount, position, size, and brightness of the RSO present in the simulated images (and thus in the corresponding mask images), may be determined by the image simulator.

[0294] Once the datasets are collected and the model is trained, inspection and verification are performed. Information including detection accuracy, precision, and inference speed is collected. The inspection may be performed through a script or other program that essentially acts as an iteration of image verification without modifying the trained model. In verification, an input image for the RSO filter is provided and the model is run through to output the resulting predicted image. This predicted image is compared to the known ground truth mask for the input image to compare the accuracy of the prediction. Various factors are tested, such as the proximity of the centers of the pixel grains representing the RSO between the actual image and the predicted image, whether the predicted image has the correct amount of RSO, and / or whether stars are being misidentified as RSO.

[0295] Next, referring to FIG. 10, an exemplary simulated image 902, mask image 904, and output image (prediction) 906 according to one embodiment are shown therein. The RSO detection model (not shown) may be the model 732 of FIG. 8.

[0296] After training, the RSO detection model may be implemented according to the method 800 of FIG. 9. The RSO detection model may include the functionality of the method 100 of FIG. 1 at 108.

[0297] The simulated image 902 is input into the RSO detection model.

[0298] The simulated image 902 may be a portion of the simulated training dataset 14 generated as a portion of the pipeline 10 of FIG. 2. The simulated image 902 may be the simulated image 320 of FIG. 4.

[0299] The mask image 904 represents the expected result of the RSO detection model when identifying RSOs in the simulated image 902.

[0300] The mask image 904 shows only the RSOs present in the simulated image 902, for example, omitting all star streaks.

[0301] The output image 906 includes the identified RSO 908. For the sake of clarity, only a single RSO 908 is labeled in FIG. 10, and this RSO is circled.

[0302] It should be understood that the RSO 908 is a point inside the illustrated circle, and the other points in the output image 906 are also RSOs. The entire circle labeled 908 itself is not an RSO.

[0303] In one embodiment, the RSO detection model includes a plurality of features. To support grayscale square images, the input tensor shape is different from that of a standard U-Net input. The RSO detection model has fewer layers than a conventional U-Net model, which advantageously contributes to the above reduction in layers and greater efficiency of the modified U-Net and the trained model output. The RSO detection model is a model with a combination of other layers (such as Conv2D and Transpose) that operates in a more efficient manner, for example, using different layers to achieve similar results without hypertrophying Conv2dTranspose. The RSO detection model has approximately 50,000 trainable parameters compared to approximately 31,000,000 that a conventional U-Net model has. The output layer for the RSO detection model is set to have a "sigmoid" activation, which results in an output that is always 0 or 1, i.e., not a "relu" with two values for each pixel in the output prediction image, i.e., on or off. The pixels in the predicted image are either part of the RSO or not part of the RSO.

[0304] The output image may be considered a large matrix of 0s and 1s. When the padding for each layer in the RSO detection model, which means there is no padding with 0s, is set to "valid", the output image is made as small as possible and retains only important information (i.e., 1s). When applying this disclosure to the RSO detection context, only the pixels that are 1 are RSOs. For a given output image, there may be a relatively small number of such 1s. Thus, when the padding is set to "valid", most of the image (all 0s) is dropped, leaving a relatively small matrix of 1s with some 0s. When the padding is set to "same", the entire matrix, including 0s, is retained, meaning the matrix is always the same size.

[0305] The loss function used above was originally weighted with respect to cross-entropy. Experiments have revealed that the focal loss function is particularly suitable when the input classes for the image segmentation model are imbalanced. In one embodiment, within the model, there is a single class, i.e., whether a pixel is an RSO or a part of an RSO. The pixel is labeled as either RSO or non-RSO with a value between [0,1] representing the reliability in the prediction.

[0306] Unlike the cross-entropy loss, the focal loss only examines positive cases and does not take negative cases into account. For example, even if there are many RSOs in an image, each RSO is just a small grouping of pixels (e.g., when there are 8 RSOs in an image, such all pixels may represent less than 2% of the image).

[0307] When selecting a dataset for validation and testing, the dataset is preferably homogeneous or taken from the same source. Isolating the test data is a priority to minimize bias from peeking at or enabling the model to be trained or tuned on the test dataset or its labels. Different datasets avoid the risk that model training only focuses on some points existing in the training set. Training on different datasets leads to the model being more generalized and thus becoming a better predictor. When the data is not time-series data, the entire range of the dataset is available for segmentation for training, validation, and testing. In embodiments where computational resources are fully utilized but the dataset is relatively large, it may be beneficial to segment the dataset into a test and training split, and further segment the training set into a training subset and a validation subset. The optimal split may be project-specific, but a common split often resembles 80 / 10 / 10 or 60 / 20 / 20.

[0308] When the training data is being fitted to a model (such as model 732), the result may indicate underfitting, overfitting, or correct fitting. Underfitting occurs when there are too few independent variables or labels compared to the amount of observations.

[0309] When a trained model (such as trained model 732) is underfitted, the trained model may miss the trends in the data. When a trained model is overfitted, the trained model loses generality by focusing on details that were specific to the provided dataset. For example, if a particular training set had only streaks of stars that indicated a particular path, the model might predict that all subsequent streaks of stars are oriented along that particular path, and if the streaks of stars are oriented in different patterns, it might not be able to convey what the streaks of stars are.

[0310] In model training, validation, and testing, it is important that the data is balanced across all samples of the population to remove bias and the possibility of overfitting. In the experiment, a single label representing the amount of RSOs present in the image, ranging from 0 to 8, was chosen. The sets were sorted and split according to a predefined split ratio (70 / 15 / 15) for training, validation, and testing respectively. The test dataset was kept completely isolated to ensure that the model did not have access to them prior to the actual test.

[0311] The training set was composed of simulated images 320 generated from the image simulator 305. None of the simulated images 320 had an overlap with RSOs and streaks of stars. In a real-world environment, such a conjunction is short-lived, and thus this corner case is not a problem. The training set included images with 0 RSOs, a single RSO, and multiple RSOs present in the simulated images 320. All images had streaks of stars and a certain level of background noise.

[0312] The validation set is a part of the training set, that is, the validation set contains pictures in the same format generated simultaneously. To validate the model against images it has not seen before, the validation set is separated from the training set before the model is trained.

[0313] The test set is composed of 320 simulated images generated as in the training set. In this experiment, the test set was taken from the same pool of the simulated images 320 as the training set. However, none of the simulated images 320 in the validation set were previously in the training set.

[0314] Referring now to FIG. 13, shown therein is a system 1300 for on-board detection of RSOs in optical images using machine learning, according to one embodiment. System 1300 may implement any one or more of the methods, systems, or devices described above.

[0315] System 1300 includes a ground segment 1302 and a space segment 1304.

[0316] The ground segment 1302 includes a ground terminal 1306. The ground terminal 1306 includes a data storage device 1308 such as a computer memory, a processor 1310 communicating with the data storage device 1308, and a communication link unit 1312. The ground terminal 1306 may include a single device or platform, or multiple devices or platforms (which may be local or remote to each other). Thus, the components 1308, 1310, 1312 may be implemented on the same device or platform, or across multiple devices or platforms.

[0317] Processor 1310 is configured to execute an image simulator module 1314 and a model training module 1316. Processor 1310 may further include a human-machine interface (not shown) to enable a user to interact with the image simulator and modules 1314, 1316.

[0318] The image simulator module 1314 is configured to generate simulated training images 1318 according to one or more methods described herein. The simulated training images 1318 are stored in the data storage device 1308.

[0319] The data storage device 1308 further stores real training images 1320. In a variant, the system 1300 may use a combination of the simulated training images 1318 and the real training images 1320 for model training, only the simulated images 1318 for model training, or only the real images 1320 for model training. In embodiments that use only the real training images 1320, the processor 1310 may not include the image simulator module 1314.

[0320] The simulated training images 1318 and / or the real training images 1320 are provided as inputs to the model training module 1316.

[0321] The model training module 1316 executes a machine learning algorithm to train an RSO detection model 1322 using the training data 1318, 1320 as described herein. The trained model 1322 output by the model training module 1316 is stored in the data storage device 1308.

[0322] The ground terminal 1306 uplinks the trained RSO detection model 1322 to the space segment 1304 via the communication link unit 1312. In one embodiment, the trained RSO detection model 1322 is implemented by uplinking the model to a satellite. In another embodiment, the trained RSO detection model 1322 is implemented by placing the model on a ground processor and further sending it along with the satellite when launched.

[0323] Next, referring to the space segment 1304, the space segment 1304 includes a spacecraft 1324. The spacecraft 1324 may be a satellite. The satellite may be part of a satellite constellation. The satellite may communicate with other satellites in the constellation via inter-satellite links (e.g., optical, RF).

[0324] The spacecraft 1324 includes an optical sensor 1326 for collecting electro-optical images, a data storage device 1328 (e.g., a data storage device, etc.), a processor 1330 in communication with the data storage device 1328, and a communication link unit 1332. The data storage device 1328 and the processor 1330 are components of an on-board computer system (sometimes also referred to as an on-board processor or OBP). The on-board computer system, and thus the data storage device 1328 and the processor 1330, may be implemented in a single computing device or across multiple computing devices.

[0325] The communication link unit 1332 receives the trained RSO detection model 1322 from the ground terminal 1306 via the uplink. In FIG. 13, the trained RSO detection model 1322 is shown as being uplinked from the ground segment 1302 to the spacecraft 1324, but in other embodiments, it should be noted that the trained RSO detection model 1322 may be uploaded from the ground terminal 1306 to the spacecraft 1324 (i.e., to the data storage device 1328 described below) within the ground segment 1302 before the launch of the spacecraft 1324.

[0326] The trained model 1322 is implemented as part of the RSO detection module 1334 executed by the processor 1330. In some embodiments, the RSO detection module 1334 may be configured to optimize the model 1322 for on-board processing.

[0327] The optical sensor 1326 is configured to collect the optical image 1336. The parameters of the imaging operation performed by the optical sensor 1326 may be determined by the processor 1330 (e.g., based on one or more commands or tasks received from the ground segment 1302).

[0328] The optical image 1336 is stored in the data storage device 1328.

[0329] The optical image 1336 is provided as an input to the trained RSO detection model 1322. The RSO detection model 1322 performs machine learning-based RSO detection as described herein. In doing so, the trained model 1322 generates RSO detection data 1338. The RSO detection data 1338 represents the RSO detected in the optical image 1336. The RSO detection data 1338 is stored in the data storage device 1328.

[0330] The RSO detection data 1338 may be further processed by the processor 1330 (e.g., by the RSO detection module 1334).

[0331] The RSO detection data 1338 is provided to the communication link unit 1332 for transmission to an external system of the spacecraft 1324, such as within the ground segment 1302 (e.g., the ground terminal 1306 or another ground terminal), regardless of whether it is raw or post - processed.

[0332] The above description provides examples of one or more systems, methods, or devices, but it should be understood that other systems, methods, and devices may fall within the scope of the claims as interpreted by those skilled in the art.

Description of Reference Numerals

[0333] 122 Foreground generator 124 Background generator 126 Image generator 300 Computer system 302 Processor 304 Memory 305 Image simulator 306 Foreground generator, foreground generator module 308 Background generator, background generator module 310 Image generator, image generator module 316 Foreground table 318 Background table 320 Simulated image 600 U - Net CNN segmentation model 602 Input layer 604 Hidden layer 610 Output layer 612 Processing node 700 Computer system 702 Low - power processing unit 704 Memory 730 Low - power GPU 732 Model, ONNX file 735 Input 902 Simulated image 904 Mask image 906 Output Image 908 Resident Space Object (RSO) 1300 System 1302 Ground Segment 1304 Space Segment 1306 Ground Terminal 1308 Data Storage Device 1310 Processor 1312 Communication Link Unit 1314 Image Simulator Module 1316 Model Training Module 1318 Simulated Training Image, Training Data 1320 Real Training Image, Training Data 1322 RSO Detection Model, Trained Model 1324 Spaceship 1326 Optical Sensor 1328 Data Storage Device 1330 Processor 1332 Communication Link Unit 1334 RSO Detection Module 1336 Optical Image 1338 RSO Detection Data

Claims

1. 1. A method for training an image segmentation model to generate simulated images for detecting resident space objects (RSOs), the method comprising: generating a foreground of the simulated image; Calculating the coordinates of an imaging satellite at a given time; determining a region of interest given a field of view (FOV) of the imaging satellite; and calculating coordinates of all RSOs and saving said coordinates if said RSOs are within said region of interest; generating a foreground comprising an RSO; generating a background for the simulated image; querying a star catalog database for coordinates of all stars that fall within said FOV; adding noise to the simulated image; and Transforming RSO coordinates and star coordinates into pixel coordinates in said simulated image; generating a background by generating the simulated image; Select an imaging mode and Setting an exposure time; and Rendering the simulated image according to the imaging mode and the exposure time. and A method comprising:

2. The method of claim 1 , wherein calculating the coordinates of all RSOs is performed using a propagation algorithm.

3. The method of claim 1 , wherein the noise comprises star streaks.

4. The method of claim 1 , wherein the imaging mode is selected from among star tracking, target rate tracking, and custom rate.

5. 2. The method of claim 1, wherein calculating coordinates of all RSOs and saving the coordinates if the RSOs are within the region of interest comprises simulating an orbital path to generate a list of access windows when a foreground satellite is within the FOV.

6. The method of claim 5 , wherein the timestamp associated with each access window is slightly randomized such that the foreground satellite is not always centered within the simulated image.

7. 6. The method of claim 5, wherein a timestamp associated with each access window is used to calculate the right ascension (RA) and declination (DEC) of all foreground satellites within the FOV.

8. The method of claim 7 , wherein transforming the RSO to the pixel coordinates in the simulated image is performed according to the RA and the DEC.

9. 1. A method for training an image segmentation model to detect resident space objects (RSOs), the method comprising: generating a training set by performing the method of claim 1 on a plurality of simulated images; training the image segmentation model using the training set and / or one or more real images; A method comprising:

10. The method of claim 9 , wherein the method is executed in a low power GPU or FPGA.

11. 1. A method for detecting a resident space object (RSO) in an optical image, the method comprising: inputting the optical image into a trained image segmentation model, the trained model having been trained using the method of claim 9; Detecting RSOs in the optical images using the trained image segmentation model; and A method comprising:

12. 1. A method for training an image segmentation model for detecting RSO, the method comprising: generating a simulated training data set, the simulated training data set including at least one simulated training image, each simulated training image simulating a view from an imaging satellite or a ground-based optical system; curating the simulated training dataset; customizing the simulated training data set for edge cases, different sensor types, and / or different imaging satellites; and Combining different simulated training datasets and curating the content by performing one or more of the following: processing the simulated training data set into a training subset, a test subset, and a validation subset; generating an RSO detection model by training a modified U-Net on the training subset; Testing the generated RSO detection model using the test subset; and validating the generated RSO detection model by inputting the validation subset into a plate solving algorithm; A method comprising:

13. 1. A method for detecting an RSO in an optical image, the method comprising: generating simulated training images using an image simulator, each of the simulated training images simulating a view from an imaging satellite or a ground-based optical sensor and including a simulated foreground including an RSO and a simulated background including stars; training an RSO detection model using the simulated training images; configuring the trained RSO detection model for on-board processing, the on-board processing being aboard a satellite; and Detecting RSOs in the optical images using the trained RSO detection model configured for on-board processing; A method comprising:

14. 14. The method of claim 13, wherein the image simulator applies a configurable scale of brightness that incorporates a pseudo-random element and produces realistic brightness variations across the simulated training images, and the size of the RSO in the simulated foreground is randomized between 3 and 5 pixels in size.

15. 1. A method for detecting an RSO onboard a satellite, the method comprising: capturing an optical image using an optical sensor on board the satellite; providing the optical image to a processor on board the satellite, the processor executing an RSO detection model trained using a plurality of simulated training images simulating a view from an imaging satellite, each simulated training image including a simulated foreground including an RSO and a simulated background including stars; Detecting RSOs in the optical image using the RSO detection model; and controlling the attitude of the satellite based on the output of the RSO detection model; A method comprising:

16. The method of claim 15 , wherein the method is executed in a low power GPU or FPGA.

Images