Systems and methods for detecting and mapping space objects

Abstract

Methods and systems of creating accurate data representations of the positions and other characteristics of target space objects using data from a plurality of space-borne artificial satellites and ground-based sensors. Such methods and systems are deployable either on a single satellite, on a plurality or swarm of satellites, and / or in a ground-based system. Various artificial intelligence and machine learning processes described herein are employed to generate a latent data representation of passively-received broad spectrum multi-modal signal data systematically integrated with advanced multi-sensor data fusion algorithms.

Description

PATENT APPLICATIONSYSTEMS AND METHODS FOR DETECTING AND MAPPING SPACE OBJECTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing of U.S. ProvisionalPatent Application No. 63 / 643,023, entitled " Systems and Methods for Detecting and Mapping Space Objects", filed on May 6, 2024, and the specification thereof is incorporated herein by reference.BACKGROUND OF THE INVENTIONTechnical Field:

[0002] Embodiments of the present invention relate to systems and methods for detecting and mapping space objects through a network of space-based and ground-based sensors, in particular, for creating accurate representations of the positions and velocities of space objects by utilizing passively-received broad spectrum multi-modal signals systematically integrated with advanced multi-sensor data fusion algorithms, leveraging artificial intelligence and machine learning paradigms.Background Art:

[0003] Space-debris poses an existential risk to the growing space infrastructure and economy in Low Earth Orbit (“LEO”) and beyond. There are estimated to be over 1 million untracked objects in LEO that are expected to grow as launch cadence and anti-satellite warfare continues, increasing the risk of the catastrophic damage to the over 20,000 satellites expected to occupy LEO by 2027. Debris smaller than 5cm cannot be detected and then tracked by existing radar and optical systems, in part because existing systems have increased loss of signal from attenuation, doppler shift, and small radar cross-section. Although the number of signals received can be increased by deploying more sensors or more expensive sensors, such methods are cost prohibitive. Furthermore, traditional data processing methods and narrow satellite communication bandwidths make processing the expanded amount of sensor data incredibly difficult.

[0004] Accordingly, what is needed is a system and method for detecting space debris that is capable of deriving more signals out of existing or inexpensive sensor systems and the ability to efficiently convert that increased amount of signals into space object positions and characteristics which may be characterized as a dynamic map of the space environment.

[0005] Note that this application may refer to other publications or existing art. Discussion of such publications or existing art herein is given for more complete background and is not to be construed as an admission that such publications are prior art for patentability determination purposes. This application refers to publications or existing art as a matter of giving a more complete background. Such references shall not be interpreted as an admission that such references are prior art for purposes of determining patentability of the present invention.BRIEF SUMMARY OF EMBODIMENTS OF THE PRESENT INVENTION

[0006] Embodiments of the present invention relate to a method of mapping space objects, where the method includes: obtaining data from a plurality of space-borne artificial satellites (“Sensed Signal Data”) and ground-based sensors (“Ground-Based Sensor Data”), each of the plurality of space-borne artificial satellites and ground-based sensors including at least one sensor receiving electromagnetic signals emitted directly by natural or artificial phenomena in space (“Emitting Space Objects”) or reflected off of other natural or artificial space objects (“Reflecting Space Objects”), the Sensed Signal Data of any given satellite of the plurality of space-borne artificial satellites and the Ground-Based Sensor Data of any given sensor of the plurality of ground-based sensors including information relating to electromagnetic signals reflected off of or emitted by at least one target space object (“Target Space Object(s)”), the Sensed Signal Data and the Ground-Based Sensor Data including data relating to a plurality of distinct frequency bands of the electromagnetic signals reflected off of or emitted by the Target Space Object(s); obtaining data relating to the positions of known or determinable sources or reflections of artificial or natural electromagnetic signals (“Existing Signal Data”); storing the Sensed Signal Data, Ground-Based Sensor Data, and Existing Signal Data; processing the Sensed Signal Data, Ground-Based Sensor Data, and Existing Signal Data to generate rough, partial, or ambiguous estimates of the position, velocity, attitude, novel views, and other data of Target Space Object(s) (“Candidate State Data”); storing the Candidate State Data; and generating a compressed latent data representation (“Latent Data”) of the stored Sensed Signal Data, Ground-Based Sensor Data, Candidate State Data, and Existing Signal Data; storing the Latent Data; and fusing the Latent Data into a latent data representation (“Fused Latent Data”); generating position, velocity, attitude, novel views, and other data (“State Data”) related to the Target Space Object(s)’s state using the Fused Latent Data. In another embodiment, the Sensed Signal Data and the Ground-Based Sensor Data include data relating to electromagnetic signals in each of the radio, microwave, infrared, visible light, ultraviolet, and x-ray frequency bands. In another embodiment, the step of processing the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data includes: obtaining data related to the state and parameters of the sensors of the space-borne artificial satellites and ground-based sensors (“Sensor State Data”); and employing a plurality of analytic algorithms determined by the format and underlying frequency of the Sensed Signal Data and Ground-Based Sensor Data to generate the Candidate State Data.

[0007] In another embodiment, the Candidate State Data includes two-line element data. In another embodiment, the sensors of the space-borne artificial satellites and ground-based sensors include optical sensors, and the plurality of analytic algorithms includes a multi-view stereo computer vision algorithm applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relate to optical wavelengths, to generate the Candidate State Data, where the Candidate State Data represents the position and velocity of the target space objects relative to the sensors of the space-borne artificial satellites and ground-based sensors. In another embodiment, the sensors of the space-borne artificial satellites and ground-based sensors include optical sensors, and the plurality of analytic algorithms includes a multi-view synthesis algorithm applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relate to optical wavelengths, to convert the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data from two- dimensional space into a three-dimensional representation to be used in the step of generating Candidate State Data. In another embodiment, the sensors of the space-borne artificial satellites and ground-based sensors include radio frequency sensors, and the plurality of analytic algorithms includes aperture synthesis algorithms applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relates to radio frequency data, to generate the Candidate State Data, where the Candidate State Data represents the position and velocity of space objects in the data relative to the sensors. In another embodiment, the sensors of the space-borne artificial satellites and ground-based sensors include radio wavelength phased arrays, and the plurality of analytic algorithms includes beamforming algorithms to spatiotempo rally optimize the received signal processing to maximize the phased array antenna's sensitivity to signals arriving from a specific direction while simultaneously suppressing interfering signals and noise arriving from other directions, the algorithms applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relates to data received from the radio wavelength phased arrays sensors, to generate the Candidate State Data. In another embodiment, the sensors of the space-borne artificial satellites and ground-based sensors include radio frequency sensors, and the plurality of analytic algorithms includes range-based, time-based, and angle-based relative localization algorithms applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that includes radio frequency data. In another embodiment, the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data further includes data that is known and data that is unknown, and the step of processing such data further includes: regarding data that is not previously known, amplifying and storing the data in a database (“Known Signal Database”); and regarding data that is known, checking such known data against data in the Known Signal Database and repeating the same for a period of time T.

[0008] In another embodiment, the step of processing the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data further includes applying a bi-static range processing algorithm and a time difference of arrival (“TDOA”) algorithm, using data obtained from the Known Signal Database. In another embodiment, the step of processing the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data further includes establishing a bi-static range ellipsoid capturing a range of possible positions of the Target Space Object and storing the same as Candidate State Data. In another embodiment, the Candidate State Data includes hyperboloids generated by the TDOA algorithm, the hyperboloids capturing a range of possible positions of the Target Space Object. In another embodiment, the method further includes combining a plurality of Candidate State Data and storing as Candidate State Data the intersection of the hyperboloids generated by the TDOA algorithm, thereby reducing the ambiguity of the resulting Candidate State Data.

[0009] In another embodiment, the method further includes obtaining time-related data(“Sensed Signal Time Data”) from each of the plurality of space-borne artificial satellites and ground- based sensors relating to the time such Sensed Signal Data or Ground-Based Sensor Data was received by the given satellite or ground based-sensor of the plurality of space-borne artificial satellites or ground-based sensors; storing the Sensed Signal Time Data; and where the steps of generating Candidate State Data, generating Latent Data and fusing the Latent Data further includes using the Sensed Signal Time Data. In another embodiment, the step of generating Latent Data includes applying a first encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the radio frequency band, a second encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the microwave frequency band, a third encoder to the Sensed Signal Data and the Ground- Based Sensor Data in the infrared frequency band, a fourth encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the visible light frequency band, a fifth encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the ultraviolet frequency band, and a sixth encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the x-ray frequency band, where each encoder includes neural network architectures. In another embodiment, the step of generating Latent Data includes applying a single encoder to the Sensed Signal Data and the Ground-Based Sensor Data, where the single encoder includes a neural network architecture. In another embodiment, the step of generating Fused Latent Data further includes: generating distinct data within the Latent Data for a particular moment in time of the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data; and applying at least one process selected of the group consisting of: concatenation, pooling, weighted mean, median, and other combination processing. In another embodiment, the step of generating Fused Latent Data further includes processing the Latent Data by attention processes selected from the group consisting of: self-attention, cross-attention, dot-product attention, and multi-head attention processes, to learn contextual parameters that optimally combine the Latent Data.

[0010] In another embodiment, the step of generating Fused Latent Data further includes using Bayesian estimation processing in the case that Latent Data relates to Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data received over a period of time. In another embodiment, the Bayesian estimation processing includes: using the Latent and Fused Data related to a first moment in time to predict Fused Latent Data (“Predicted Fused Latent Data”) related to asecond moment in time. In another embodiment, the Bayesian estimation processing includes: relating the Predicted Fused Latent Data of the second moment in time to the Fused Latent Data of the first moment in time using a recursive fusion filtering process selected from the group consisting of: Kalman filtering, Kalman net and particle filers. In another embodiment, the step of using the Latent and Fused Data related to the first moment in time to the Predicted Fused Latent Data related to the second moment in time further includes: modeling the Predicted Fused Latent Data as a hidden state of stored data within a recurrent neural network, the model evolving based on inputted Latent and Fused Latent Data related to the first moment; and modelling within the recurrent neural network the covariance of error between the Predicted Fused Latent Data and an actual Fused Latent Data based on a plurality of Latent Data and Predicted Fused Latent Data; projecting the hidden state to a Kalman gain; and updating the current time Fused Latent Data to be the sum of the Predicted Fused Latent Data and the Kalman gain multiplied by the difference between the Predicted Fused Latent Data and the actual Latent Data. In another embodiment, the recurrent neural network is a reservoir computer.

[0011] In another embodiment, the step of generating State Data includes applying a decoder or plurality of decoders (“decoder(s)”), where each decoder includes a neural network architecture or plurality of neural network architectures. In another embodiment, the decoders generate visual representations of the Fused Latent Data in the form of a range-doppler map. In another embodiment, the decoders generate visual representations of the Fused Latent Data in the form of fixed frames of reference to the earth. In another embodiment, the step of storing the Sensed Signal Data, Ground-Based Sensor Data, Candidate State Data, and Existing Signal Data includes storing the same in a first data storage device disposed on a first satellite of the plurality of space- borne artificial satellites, the first data storage device including the instructions for generating the Latent Data. In another embodiment, the step of generating the State Data is performed by instructions stored in a data storage device disposed on Earth. In another embodiment, the step of storing the Sensed Signal Data, Ground-Based Sensor Data, Candidate State Data, and Existing Signal Data includes storing the same in a first data storage device disposed on Earth, the first data storage device including the instructions for generating the Latent Data. In another embodiment, the State Data also represents the albedo, shape, or attitude of the Target Space Object(s). In another embodiment, the State Data generated is a three-dimensional representation of the relative position and velocity of Target Space Object(s) within a certain distance of a satellite of the plurality of space- borne artificial satellites. In another embodiment, the method further includes a first satellite of the plurality of space-borne artificial satellites transmitting its Sensed Signal Data or Latent Data to a second satellite of the plurality of space-borne artificial satellites.

[0012] Objects, advantages and novel features, and further scope of applicability of the present invention will be set forth in part in the detailed description to follow, taken in conjunction with the accompanying drawings, and in part will become apparent to those skilled in the art uponexamination of the following, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one or more embodiments of the present invention and, together with the description, serve to explain the principles of the invention. The drawings are only for the purpose of illustrating one or more embodiments of the invention and are not to be construed as limiting the invention. In the drawings:Fig. 1 is a schematic that illustrates the EM signals a satellite receives and employs to detect and map the position and other aspects of debris relative to the satellite, using signals of opportunity from other satellites in the neighborhood and reflections thereof, according to an embodiment of the present invention;Fig. 2 is a schematic that illustrates a plurality or swarm of satellites in communication with each other to exchange data related to the maps of each satellite’s sensing neighborhood so that any one of the satellites can create a map of space debris that extends beyond the neighborhood of that particular satellite, according to an embodiment of the present invention;Fig. 3 is a schematic that illustrates a flow chart of a process employed to transform signals from the multitude of EM sensor data into a sensing satellite space object map, according to an embodiment of the present invention;Fig. 4 is a schematic that illustrates space object EM data undergoing processing with a centralized sensor fusion architecture and a pre-processing component, according to an embodiment of the present invention;Fig. 5 is a schematic that illustrates space object EM data undergoing processing with a distributed sensor fusion architecture and a pre-processing component, according to an embodiment of the present invention;Fig. 6 is a schematic that illustrates signal of opportunity EM data captured from an antenna sensor to be processed by range-based, time-based, and angle-based relative localization algorithms and is employed by processors onboard a sensing satellite, according to an embodiment of the present invention;Fig. 7 is a schematic that illustrates how the candidate states of the space object from the preprocessing component are packaged together with the available space object EM data and signal of opportunity EM data, including the data already processed by the pre-processing component and processed further within components of a multi-modal model, according to an embodiment of the present invention;Fig. 8 is a schematic that illustrates some subsystems used with latent space sensor fusion to fuse a multitude of data in latent space representation, according to an embodiment of the present invention;Fig. 9 is a schematic that illustrates how swarm embodiments can subscribe to a distributed architecture and be realized in the form of a swarm, according to an embodiment of the present invention;Fig. 10 is a schematic that illustrates how edge server embodiments can be realized in the form of a network of controllable cooperative edge server sensing satellites, according to an embodiment of the present invention;Fig. 1 1 is a schematic that illustrates a system that is realized in the form of pure software running on data processing hardware that collects space object EM data and signal of opportunity EM data from the ground, air, and space, according to an embodiment of the present invention; andFig. 12 is a flow diagram that illustrates how data can be sensed, processed, encoded and fused, according to an embodiment of the present invention.DETAILED DESCRIPTION OF THE INVENTION

[0014] As used herein, “space” means anything in the physical universe beyond earth’s atmosphere and / or anything beyond the Karman line, including low earth orbit (“LEO”). The term “space-borne” means that the subject object is disposed in space. The phrase “space object” shall mean any object or phenomena located in space, artificial or natural. The word “satellite” means any artificial satellite or spacecraft, of any size, purpose or use, unless otherwise stated or implied from context to refer to a natural satellite. The word “neighborhood”, when used in context of a satellite, means the space within the range of a particular satellite’s sensors. The term “ground”, for example as used to describe a ground-based object, means that subject object is disposed on the terrestrial surface of a planet and / or within the atmosphere of a planet. The term “ground station” refers to a ground-based satellite communication system that is connected to centralized computing infrastructure through terrestrial networks. The acronym “EM” refers to “electromagnetic”. The term “frequency” is used interchangeably with the term “wavelength”. As used herein, the term “band” isalso used interchangeably with “frequency” and “wavelength” and should be interpreted to carry the same meaning unless the context otherwise demands. The terms “map” and “representation” are used interchangeably herein and is not intended to be limited by any particular type of means of display; the terms “map” and “representation” should be interpreted to include any data that, when further processing is performed, can reveal information about a target space object, including information about its position, velocity, and other characteristics. The term “swarm”, when used in the context of satellites, shall mean a plurality or more than one satellite which can be interconnected through inter-satellites communication links. The term “signal EM data” means any data related to an EM signal, including space object EM data 112, signal of opportunity EM data 116, whether known or unknown. The term “data” means any information that can be stored in a medium, for example information on which operations may be performed by a computer processor, information stored and transmitted in the form of electrical signals recordable on magnetic, optical, or mechanical recording media. The terms “received” and “sensed” are used interchangeably and should be interpreted to have the same meaning unless the context requires otherwise. The term “control system” means any computer or computer processors and associated data storage devices comprising algorithms and / or instructions for the processors to perform processes related to the satellite, including to receive guidance commands, process sensor feedback, and generate control signals to manipulate the spacecraft's actuators (e.g. thrusters or reaction wheels), to achieve and maintain the desired orientation, position, and velocity of the satellite or other satellite tasks including, but not limited to actuator, attitude, and orbital control, thrust vectoring, pointing, real-time adjustments, and enacting the maneuvers and trajectories of the guidance and control systems.Overview

[0015] Embodiments of the present invention are directed to systems and methods that can be employed on sensing satellite 100 or at ground-based system 101 for detecting and mapping the position of space object(s) 30, for example space debris. Fig. 1 illustrates an example of how sensing satellite 100 employing the system of the present invention may detect and determine the position of space object 30 within its sensing neighborhood 111. Sensing satellite 100 is in orbit around planet 1 , for example in low earth orbit (LEO) around earth. As is more common as the space economy grows, neighborhood 111 of sensing satellite 100 may contain space object(s) 30, which may be of any size and quantity. Other satellites, for example satellite 10 and satellite 20, may also be positioned in the neighborhood of sensing satellite 100 or at least emitting electromagnetic signals 12 and 22 into the neighborhood of sensing satellite 100 even if not positioned within the neighborhood 111. Satellites 10 and 20 may have known or determinable positions and / or known or determinable electromagnetic signals 12 and 22, which known or determinable positions may be stored in a database on earth or in a database on sensing satellite 100 (the database referred to herein as existing signal database 40). Satellites 10 and 20 are likely to emit electromagnetic signals 12 and 22 respectively, which may reflect off of space object 30 as space object EM signals 32. Space object EM signals 32 can then besensed by sensors 110 disposed on sensing satellite 100 or by ground-based sensors 110 if such are employed in the particular application. The earliest received of each signal, for example electromagnetic signals 12 and 22, are also sensed by sensors 110 and stored as signal of opportunity EM data 116 in known signal database 40. As illustrated in Fig. 12 broadly illustrating the data processes of the present invention, data related to the space object EM signals 32 is stored as space object EM data 112 in a data storage device 114 disposed either on sensing satellite 100 or at ground-based system 101. Space object EM data 112 preferably comprises data relating to the electromagnetic signals for each frequency of space object EM signals 32.

[0016] Software instructions 120 stored in data storage device 114 on sensing satellite 100, or as stored on data storage devices disposed in other systems (for example, ground-based system 101), comprise instructions that when executed will employ the various processes described throughout this application to generate a representation that can be ultimately interpreted as space object map 150. Details of the foregoing systems and methods will be further described herein, including particular methods and systems for employing artificial intelligence and data fusion to fuse a multitude of data in a multi modal model 444. For example, referring to Fig. 12, space object EM data 112 and / or signal of opportunity EM data 116 is preferably subjected to “pre-processing” (for example pre-processing component 456) to amplify signals and / or determine candidate solutions 113. For another example, referring to Fig. 4, space object EM data 112 and / or signal of opportunity EM data 116 that has already been preprocessed is encoded into latent space representation 124 as will be described in further detail, for example as illustrated in Fig. 3. The data is then fused to generate or otherwise decoded to generate space object map 150 as will be described in further detail.

[0017] Embodiments of the present invention are also directed to systems and methods for creating a map of space objects created from a plurality of satellites and / or ground-based sensors, some versions of which may be referred to herein as the “swarm” versions. Fig. 2 illustrates an example of how any one sensing satellite 100a, 100b, 100c, 100d of a plurality of sensing satellites 100, each comprising the system of the present invention described in the paragraph above for sensing satellite 100, receives data 152 (for example latent space representation 124 and / or latent space fused data 125) of any or every other sensing satellite 100 to create a space object map comprising at least a portion of each other sensing satellite’s space object map 150. In this way, a space object map larger than the area of space in any one satellite’s neighborhood 151 can be created, potentially for an entire orbit assuming there are enough sensing satellites with sensing neighborhoods that cover the entire orbit.

[0018] Further details regarding the above systems and methods are described below, as well as numerous other embodiments.Sensing Electromagnetic Signals of Space Objects and Storing and Processing Related Data

[0019] Space objects, whether the space object is the space object 30 to be tracked or other space objects such as satellites 10 and 20, have an electromagnetic signature, either by virtue of generating a source of electromagnetic signals or reflecting or causing disturbances in existing electromagnetic fields. Embodiments of the present invention comprise or employ sensors 110 capable of receiving electromagnetic signals emitted directly by natural or artificial space objects or reflected off of natural or artificial space objects. Sensors 110 may be disposed on sensing satellite 100 or at ground-based system 101 , which may comprise a single sensor 110 or any number of sensors 110. Sensors 110 are capable of receiving any electromagnetic signal, and more preferably anything including and between the radio wavelengths and X-ray wavelengths of EM radiation.Sensors 110 can include, but are not limited, to optical wavelength Electro-Optic (“EO”) detectors, Infrared (“IR”) detectors, broadband phased array Radio Freguency (“RF”) antennas, or a combination thereof.

[0020] Space objects 30 can be described as having space object state(s) 222 comprising position, velocity, and other orbital parameters, including but not limited to, attitude, attitude rates, and other geometric attributes with respect to different geometric frames that allow for defining positions and velocities with respect to Earth, other space objects, and other frames of reference.

[0021] Electromagnetic signals (for example, 12 and 22) reflected off of a space object, for example reflected off of space object 30 as space object EM signals 32, are sensed by sensors 110 and stored as space object EM data 112 in data storage device 114.

[0022] Sensor 110 can collect sensor state data 333. Sensor state data 333 can include the complete or partial, potentially noisy or imprecise, position, velocity, attitude, attitude rates, and other geometric attributes of sensor 110 with respect to different geometric frames that allow for defining motion and orientation with respect to Earth, other space objects, and other frames of reference in data storage device 114.

[0023] The systems and methods of the invention can be employed either on sensing satellite 100, a plurality of sensing satellites 100 or on ground-based system 101. For example, in one embodiment, data storage device 114 can be disposed on sensing satellite 100 within the control system of sensing satellite 100 or be additional to (for example, a retrofit to) any storage device within the control system, depending on the particular application. In another embodiment, data storage device 114 may be disposed on ground-based systems 101 , for example as illustrated in Fig. 11.

[0024] Space object EM data 112 may be of any format or data type as generated by corresponding sensors 110. It comprises a structure that permits the separate use and analysis of each frequency, wavelength or band of space object EM signals 32.

[0025] Sensors 110 also sense electromagnetic signals emitted by space objects, for example, electromagnetic signals 12 and 22 of satellite 10 and satellite 20, which electromagnetic signals are stored in data storage device 114 as signal of opportunity EM data 116.

[0026] Signal of opportunity EM data 116, which may be a subset of space object EM data112, may be of any format or data type. Signal of opportunity EM data 116 comprises a known or determinable structure, including as frequency, modulation scheme, coding scheme, periodicity, and other attributes that permit the identification and isolation from space object EM data 112. Signal of opportunity EM data 116 preferably permits separate use and analysis of each frequency, wavelength, or band of that collection of electromagnetic signals of the space objects to algorithmically determine the time and states, such as position and velocity, of the space objects emitting signal of opportunity EM data 116. As such, these can be used to reconstruct space object states 222, including its position and velocity.

[0027] Note that while Fig. 2 illustrates two satellites 10 and 20 emitting electromagnetic signals 12 and 22 respectively, embodiments of the present invention can operate with any number of such electromagnetic signals. Such space objects need not be artificial satellites as Fig. 2 illustrates, but can also be natural phenomena, natural space objects, other sensing satellites 100, or even space debris the position of which has already been determined or which can be determined.

[0028] While space object EM data 112 and signal of opportunity EM data 116 are referred to herein as separate objects for purposes of describing various embodiments of the present invention, in some embodiments they can be the same data. In practice, EM signals sensed by sensing satellite 100 may be received together and accordingly may be stored as a single data set of signal EM data. Any reference to space object EM data 112 and signal of opportunity EM data 116 shall not necessarily be interpreted to be completely separate sets of data or exclusive of the other, but should be interpreted to refer to subsets of data within a single set of data (e.g., signal EM data) unless the context requires otherwise.

[0029] Space object EM data 112 and signal of opportunity EM data 116 preferably each comprise time data related to the moment in time sensor 110 received a particular electromagnetic signal (e.g., 12, 22 or 32 as the case may be). Each of space object EM data 112 and signal of opportunity EM data 116 may have its own time data because the difference in the time space object EM signal 32 is received and the time an electromagnetic signal of opportunity such as electromagnetic signal 12 is received can be used to determine the position of space object 30. Tothat end, sensing satellite 100 may comprise any number of time clock generators to accurately stamp space object EM data 112 and signal of opportunity EM data 116 with the time they were received / sensed by sensor 110. The time clock generators may incorporate distributed synchronization schemes or may synchronize to a common clock, such as the Global Positioning System clock time.

[0030] Referring to Fig. 3, in another embodiment, the processors on sensing satellite 100 or on a ground-based system 101 may execute software instructions 120 comprising an ensemble of variational autoencoders (“VAE”) 122. Each VAE 122 in the ensemble processes signal EM data from frequency bands of input signals received by sensing satellite 100, and in a manner unique and appropriate to each of the frequency bands, compresses these into latent space representation 124 where critical features from the neighborhood are represented. This latent space representation 124 may be common and shared among VAEs.

[0031] Latent space representation 124, for which all frequencies of signal EM data are mapped or related to, may be chosen such that space objects 30 and satellite motion is consistent with physical laws but also representative of albedo, shape, and other attributes which can be extracted from all frequencies of collected EM signals.

[0032] Latent space representations 124 of the space objects 30 from each of the frequency bands may be processed further, for example by feeding it into / applying a self-attention mechanism including but not limited to a scaled cosine similarity, which correlates and fuses the different features in latent space representations 124 to resolve object position, orientation, and other ambiguities in sensing satellite 100 neighborhood. As used herein, the term “fuse” has the meaning a person of ordinary skill in the art of data science would understand that term to mean, including to relate such data in a manner that the data is combined so as to improve its quality.

[0033] Latent space representation 124 is processed further, for example by feeding it into / applying decoder 126, which can be artificial neural network decoder or novel view synthesizer which transforms it into sensing satellite space object map 150, which is understandable or recognizable to a human or machine system, including but not limited to as a displayable image or 3D map.Pre-processing EM Signal Data

[0034] Figs. 4 and 5 schematically illustrate some of the processes of the systems and methods of the present invention performed to process data received from sensors 110 into data that can be used to generate data representing the location and other attributes of space objects 30, asperformed by processors applying to data stored on data storage devices disposed either on sensing satellite 100 or on the ground. Various methods of doing so are further described herein.

[0035] As illustrated in Figs. 4 and 5, signal EM data 112 can undergo processing by a multitude of analytic algorithms within pre-processing component 456 determined by the format and the underlying EM signal frequency and type of sensors 110 alongside sensor state data 333 and sensor parameters data 334 to detect, estimate, and output candidate solutions 113 of space object state 222 in the data, including in the form of relative position and velocity data. The number of sensors 110 which provides signal EM data can be any number from 0 to N, where N is the total number of sensors 110 within the system.

[0036] Embodiments of pre-processing component 456 may follow centralized data flow architectures, as illustrated in Fig. 4, wherein space object EM data 112 and signal of opportunity EM data 116 from all of the N sensors 110 alongside sensor state data 333 and sensor parameters data 334 are stored in data storage device 114 and processed accordingly.

[0037] Other embodiments may follow distributed data flow architectures, as illustrated inFig. 5, wherein a multitude of disparate data storage devices 114 exist and a subset of sensors K<N are locally accessed by a plurality of pre-processing components 456 depending on whether or not they have shared access to the same data storage device 114.

[0038] Subscribing to the centralized architecture of Fig. 4, there can be a singular preprocessing component 456 that can access all of space object EM data 112 and signal of opportunity EM data 116 corresponding to the set of sensors 110. Within the pre-processing component 456 a multitude of preferred algorithms are applied depending on the type of sensors 110 and may share data arbitrarily to improve their candidate solutions 113.

[0039] Subscribing to the distributed architecture of Fig. 5, the pre-processing component456 may share its candidate solution qawith other distributed instances of pre-processing components 456, while also receiving and processing their candidate solutions Qa. Within the distributed pre-processing component 456 a multitude of preferred algorithms are applied depending on the type of sensors 110 and may share data with the plurality of pre-processing components 456 to improve their candidate solutions 113 if they are a neighbor.

[0040] Space object EM data 112 from sensors 110 in the optical wavelengths are preferably processed by multi-view stereo computer vision algorithms alongside sensor state data 333 and sensor state parameters 334 to estimate the position and velocity of space objects in the data relative to sensors 110, which are characterized as candidate solutions 113. Sensor 110 may comprise a camera sensor.

[0041] Space object EM data 112 in the optical wavelengths from sensors 110 in the optical wavelengths are preferably processed by multi-view synthesis algorithms, for example neural radiance field and / or gaussian splatting, to convert the two-dimensional space object EM data 112 into a three-dimensional representation which can be used to estimate candidate solutions 113.

[0042] Space object EM data 112 from sensors 110 in the radio wavelength captured are preferably processed by aperture synthesis algorithms alongside sensor state data 333 and sensor state parameters 334 to estimate the position and velocity of space objects in the data relative to sensors 110, which are characterized as candidate solutions 113. Sensor 110 may comprise an RF antenna.

[0043] Sensors 110 including radio wavelength phased arrays producing space object EM data 112 can be processed by beamforming algorithms alongside sensor state data 333 and sensor state parameters 334 to spatiotempo rally optimize the received signal processing to maximize the phased array antenna's sensitivity to signals arriving from a specific direction while simultaneously suppressing interfering signals and noise arriving from other directions, which are characterized as candidate solutions 113.

[0044] Referring to Fig. 6, in some embodiments of the pre-processing component 456 in centralized architecture Fig. 4 and distributed architecture in Fig. 5, radio wavelength space object EM data 112 and signal of opportunity EM data 116 captured from RF antenna sensors 110 are processed by range-based, time-based, and angle-based relative localization algorithms, depending on the availability of data in known signal database 40 and sensor state and parameter data 333 and 334, respectively. In such embodiments, if the RF signals data is not previously known it is amplified and added to the known signal database 40 through port 1. If a signal is known, by checking the RF signal against the known signal database 40 through port 2, and it is repeated within some period T, where T represents a certain period of time, this corresponds to an echo and can be processed through bi-static range processing algorithms using known transmitter 156 position and velocity data from signal database 40 accessed through port 2 as well as the receiver 154 position and velocity data accessed through port 3. The signal is also processed by time difference of arrival (“TDOA”) algorithms which utilize the sensor state and parameter data 333 and 334 accessed through port 3, as well as the time difference in arrival, with respect to a global clock, of a signal measured by spatially separate receivers 154 to estimate the position and velocity of space object 30. If the receiver 154 antenna is a phased array, then angle of arrival measurements of signals across the antenna array, characterized by sensor state and parameter data 333 and 334 accessed through port 3, can be used to estimate the position and velocity of space object 30.

[0045] Such embodiments of pre-processing 456 are employed by processors onboard sensing satellite 100 or ground-based system 101 wherein through each approach the space object state 222 can be exactly estimated or ambiguities corresponding to the quantity of multiple candidate solutions 113 can be reduced.

[0046] Referring to Fig. 6 the difference in the travel time of space object EM signal 32 and signal of opportunity EM data 116 is received given that the position of satellite 10 that created signal of opportunity EM data 116 is already known and is used to establish a bi-static range ellipsoid capturing all possible positions of space object 30 relative to sensor 110 as candidate solutions 113.

[0047] Referring to Fig. 6, the bi-static range ellipsoids between a plurality of sensors 110 and space object 30 can be combined and their intersections can be attributed to relative positions of space object 30 via multi-static ranging algorithms such as trilateration, which comprises candidate solutions 113.

[0048] Referring to Fig. 6 the TDOA of signal of opportunity EM data 116 is preferably used to construct hyperboloids capturing all possible positions of space object 30 relative to sensor 110 which comprises candidate solutions 113.

[0049] Referring to Fig. 6, TDOA hyperboloids corresponding to a plurality of sensors 110 and space object 30 can be combined and their intersections can be attributed to relative positions of space object 30 via synchronous or asynchronous angular only disambiguation, which comprises candidate solutions 113.

[0050] Referring to Fig. 6, in another embodiment of the distributed architecture, two or more pre-processing components 456 computing TDOA corresponding to distinct sensors 110 share and combine their hyperboloids reducing the ambiguity of the resulting candidate solution as compared to each individually.

[0051] In some embodiments of pre-processing component 456, orbital parameters data, such as the two-line element format, and other known space object state 222 data, including but not limited to space object map 150, may be used by algorithms to reduce the ambiguity of, refine the accuracy of, or otherwise improve the candidate solutions 113.

[0052] For example, if an EM signal sensed by sensing satellite 100 is unknown but can be stored as data in data storage device 114, then the present invention can check to see if the same EM signal is sensed by sensing satellite 100 at a later time. If it is confirmed to be the same signal (e.g., by methods of time-series analysis such as matrix profile) then the processor on board sensing satellite 100 (or disposed on earth) employs various methods to reduce the ambiguity in the possibledirections and range from where the unknown EM signal came from, thereby creating a list of candidate solutions 113. If the list of candidate solutions 113 for the received EM signals corresponds to positions of known EM signals (e.g. , any known portion of signal of opportunity EM data 116, space object EM data 112, or data related to EM signals from other reflectors), then these can be eliminated to further reduce the number of possible candidate solutions in the list of candidate solutions 113.

[0053] The list of candidate solutions 113 is preferably stored in data storage device 114 to be queried later.Multi-Modal Model (MMM) and Sensor Fusion

[0054] The diversely sourced, formatted, and pre-processed space object EM data 112 is preferably processed by a neural network architecture referred to herein as the Multi-Modal Model 444 which transforms the distinct features of space object data from the different modalities of EM sensor data into a special mathematical space, the latent space, where it can be processed further to be amplified and fused .

[0055] As illustrated in Fig. 7, the candidate solutions 113 of space object state from preprocessing component 456 are then packaged together with the available space object EM data 112 and signal of opportunity EM data 116, including the candidate solutions 113 already processed by pre-processing component 456 and processed further within components of multi-modal model 444 which outputs a list of space object states 222.

[0056] A multi-modal model 444 comprises encoder 457, latent space sensor fusion 445, and decoder 458 components.

[0057] The encoder 457 and decoder 458 components of multi-modal model 444 preferably comprise neural network architectures, including but not limited to feed forward, recurrent, echo state, and other neural networks.

[0058] As illustrated in Fig. 7, multi-modal model 444 may comprise an ensemble of encoders 457 each responsible for transforming a particular type of space object EM data 112 and signal of opportunity EM data 116 as well as candidate solution 113 into their latent space representation 124.

[0059] As illustrated in Fig. 4, in embodiments subscribing to the centralized architectures, there can be a central multi-modal model 444 with latent space representation 124 that is unified for all sensors 110.

[0060] As illustrated in Fig. 5, in embodiments subscribing to the distributed architectures, a subset k of N sensors 110 feeds their data into a distributed instance comprising a multi-modal model 444 pre-processing component 456. Similar to the distributed pre-processing component 456, this distributed instance of the multi-modal model 444, and in particular the latent space sensor fusion component 445, can share its latent space representations 124 and latent space fused data 125 with the latent space sensor fusion component 445 of other distributed instances, which may in turn share their own latent space data 124,125.

[0061] The properties of latent space representation 124 can facilitate further signal processing on these features, including detection, amplification, rejection, isolation, and combinations of data. These, and other signal processing, occur within latent space sensor fusion 445 which takes as input a plurality of data in latent space representation 124 and processes these into an output that can be a singular data in the latent space representation 124, henceforth known as the latent space fused data 125, that has improved quality, such as information entropy or accuracy. Latent space fused data 125 is generated through a process that may be referred to herein as “latent space fusion” and occurs within the latent space sensor fusion 445 component.

[0062] Space object EM data 112 and signal of opportunity EM data 116, corresponding to a single time, may be transformed by multi-modal model 444 into distinct data within latent space representation 124 wherein concatenation, pooling, weighted mean, median, and other combination processing is applied within latent space sensor fusion 445 to fuse this multitude into a singular representation latent space fused data 125 for that time step.

[0063] The multitude of latent space representation 124 data corresponding to a single moment or window of time may be processed by attention mechanisms, including but not limited to self-attention, cross-attention, dot-product attention, and multi-head attention to learn contextual parameters that optimally combine the multitude of latent space representation 124 data.

[0064] The multitude of sensor data and candidate solutions 113 in latent space representation 124 corresponding to the distinct sensors 110 retrieved over a window of sequential time can be processed through recursive Bayesian estimation processing within latent space sensor fusion 445, which follows a two-step prediction and update processing. First, a prediction system uses the latent space fused data 125 from the previous time step to predict the latent space fused data 125 which is projected to the latent space representation 124. Then the predicted latent space fused data 125 is refined in the update system by adding a term that combines its projection to the latent space representation 124 with the transformed sensor data 112,116 and candidate solutions 113. This processing includes but is not limited to the methods of Kalman Filtering, KalmanNet, and particle filers that recursively fuse the multitude of latent space representation 124 data t into a singular representation for the next sequential time step, latent space fused data 125.

[0065] Referring to Fig. 8, in an embodiment of the recursive Bayesian estimation processing form of latent space sensor fusion 445, the multitude of data in latent space representation 124 corresponding to the distinct sensors 110 retrieved over a period of sequential time are processed by the prediction and update processing systems, are realized as distinct Reservoir Computer recurrent neural networks, notated in Fig. 8 as Update Reservoir Computer and Prediction Reservoir Computer. I The Prediction Reservoir Computers models the predicted temporal evolution of the latent space fused data 125 as a hidden state within the Reservoir which evolves based on inputted latent space fused data 125 from previous times. The predicted latent space fused data 125 is subsequently processed by a Readout which projects the predicted latent space fused data 125 into the latent space representation 124. The Update Reservoir Computer models the future temporal evolution of the error covariance between predicted and actual latent space fused data 125 based on the multitude of latent space representation 124 data and the predicted latent space fused data 125. The Update Reservoir Computer is processed by a readout which projects the hidden state to the Kalman gain. The current time latent space fused data 125 is updated to be the sum of the predicted latent space fused data 125 summed with the Kalman gain multiplied by the difference between the predicted latent space representation 124 and the actual latent space representation 124.

[0066] In embodiments subscribing to the centralized architecture as in Fig. 7, latent space sensor fusion 445 component can process all space object EM data 112 and signal of opportunity EM data 116 in latent space representation 124 corresponding to a centralized multi-modal model 444.

[0067] In embodiments subscribing to the distributed architecture as seen in Fig. 5, there can be a multitude of latent space sensor fusion 445 components corresponding to a plurality of multimodal model 444 components.

[0068] The outputs of multi-modal model 444 can be produced by decoder components which transform the latent space fused data 125 output of latent space sensor fusion 445 components into a plurality of outputs including space object state 222 estimating the multitude of attributes for each space object state 222.

[0069] There can be one or more decoders 458 that each transform the output of latent space sensor fusion 445 components into one or more attributes of space object state 222, including a list of orbital elements or a list of synthetic images of space object 30.

[0070] In embodiments subscribing to the centralized architecture, the list of outputs from centralized latent space fusion 445 components can be decoded by one or more decoders into one or more space object state 222 attributes for all space object 30 covered by signal of opportunity EM data 116.

[0071] One or more multi-modal model 444 decoders 458 can transform the data from latent space fused data 125 into range-doppler maps, novel views for IR and optical virtual sensors, and other attributes data of space object 30 besides space object state 222.

[0072] In embodiments subscribing to the distributed architecture, the latent space fused data 125 outputs from the one or more latent space fusion 445 components can be decoded by one or more decoders into one or more space object state 222 attributes with respect to sensors 110 or earth-centered or other fixed frames for a subset of space object 30 covered by a subset of signal of opportunity EM data 116 accessible by multi-modal model 444.

[0073] In the various manners described above, embodiments of the present invention create a scene representation of the space sensed by sensing satellite 100 by fusing, combining, synthesizing, overlaying, overlapping, integrating or otherwise using the data present in each of the different frequencies of space object EM data 112 and / or each of the different frequencies of signal of opportunity EM data 116 to create a representation or map of space object 30 relative to the sensing satellite, which map can provide additional information related to the shape, albedo, orientation, direction, distance, velocity and material of space object 30.Swarm Embodiments

[0074] Preferably, embodiments of the present invention employ data (including space object EM data 112, signal of opportunity EM data 116, candidate solutions 113 and / or latent space representation 124) from a plurality of sensing satellites 100, either as communicated between each of the satellites of the plurality of sensing satellites 100 or as communicated down to a central database disposed on earth.

[0075] For example, swarm embodiments, such as illustrated in Fig. 9, can subscribe to the distributed architecture, and be realized in the form of a swarm. A swarm is a network of cooperative sensing satellites 100a, 100b, and 100c whose sensing (e.g., beamforming) and mobility (e.g., attitude adjustment) can be coordinated through wireless communication during operation. Each swarm satellite houses the system components and data storage devices described of the invention that employ processes as described throughout this application, including but not limited to, a diverse set of sensors 110, pre-processing component 456, multi-modal model 444, corresponding latent space sensor fusion 445, or a combination thereof.

[0076] Referring to Fig. 9, in swarm embodiments, signal of opportunity EM data 116 from illuminators of opportunity (for example, electromagnetic radiation from sun 2), radio signals from LEO satellite communications (for example, radio frequency signal of opportunity 912 in the K-band), orsolar irradiation visible light that is reflected by space object 30, herein known as reflected visible light 914 are collected by sensors 110 onboard one or more satellite vehicles and by means of intersatellite (e.g., vehicle-to-vehicle (“V2V”)) links and ground links the individual satellite vehicles are able to exchange data 152 comprising the outputs of pre-processing components 456 and latent space sensor fusion 445 components, facilitating detection, isolation, and estimation of space object state 222 and other attributes through distributed multi-modal model 444 components relegated to each of the satellite vehicles in the swarm.

[0077] In swarm embodiments, the algorithms in distributed pre-processing components456, such as beamforming or aperture synthesis, can be coordinated to optimize the data processing of space object EM data 112 and signal of opportunity EM data 116 data from a particular region of space, improving the processing performance of distributed multi-modal model 444 and the quality of its space object state 222 output data.

[0078] Embodiments of the present invention are also directed to using sensing satellite space object map 150 of each sensing satellite 100 of a plurality of sensing satellites 100 to create a space object map that is referred to herein as swarm satellite space object map 160. Swarm satellite space object map 160 comprises at least a portion of sensing satellite space object map 150 or the data related thereto (for example latent space representation 124), of each sensing satellite 100 of a plurality of sensing satellites 100. In this way, a map of space objects for an entire orbit, or at least a volume of space larger than satellite neighborhood 151 of a single sensing satellite 100, can be created, which will aid in predicting the dynamics of space objects / debris.

[0079] In another embodiment, each sensing satellite 100 of a plurality of sensing satellites100 comprise receiver 154 and transmitter 156 capable of transmitting data. A first sensing satellite, for example sensing satellite 100a illustrated in Fig. 2, transmits data 152 related to its sensing satellite space object map 150 to a second sensing satellite, for example sensing satellite 100b, using its transmitter 156, and also receives, through its receiver 154, data 152 related to sensing satellite space object map 150 of a second sensing satellite, for example sensing satellite 100b. First sensing satellite 100a then, as described above of the processes it uses to make its own sensing satellite space object map 150, fuses, combines, synthesizes, overlays, overlaps, integrates or otherwise uses at least a portion of the data present in the received data 152 to refine the map of satellite neighborhoods 151 of both sensing satellites 100a and 100b.

[0080] In another embodiment, each sensing satellite 100 of a plurality of sensing satellites100 uses its transmitter 156 and receiver 154 to transmit and receive, respectively, its latent space representations 124 with other sensing satellites 100a and 100b. Each sensing satellite shares through its communication system its current latent space representation 124 to neighboring satelliteswhich each then fuse this with their own latent space representation 124, refining the representation and resolving ambiguities.

[0081] As seen in Fig. 9 in another embodiment, latent space representation 124, latent space fused data 125 and other data 152 are passed to ground-based stations 101 wherein further processing by latent space sensor fusion 445 and decoders 458 is performed to further enhance the quality of the outputted space object state 222.Edge Server Embodiments

[0082] Edge server embodiments, such as illustrated in Fig. 10, may comprise a network of controllable (e.g., altitude adjustment) cooperative edge server sensing satellites 100 each housing pre-processing component 456, multi-modal model 444, and corresponding latent space sensor fusion 445 while one or more sensors 110 can be housed within independent sensing satellites 100 that can share space object EM data 112 and signal of opportunity EM data 116 with the edge server satellites. Alongside space object EM data 112, the independent satellites housing sensors 110 may also share their sensor state data 333 as well as sensor parameters data 334 which correspond to the performance parameters such as field of view, imager resolution, beam scanning rate, and other pertinent descriptive parameters of the sensor and meta data for sensors 110.

[0083] In edge server embodiments, signal of opportunity EM data 116 from an illuminator of opportunity (for example, electromagnetic radiation from sun 2), radio signals from LEO satellite communications, or visible light solar irradiation reflected by space object 30 can be collected by sensors 110 onboard one or more independent sensing satellites 100 and by means of inter-satellite (e.g., V2V) communication. The individual satellite vehicles can exchange space object EM data 112 with the edge server satellite which can process it first through pre-processing component 456, into multi-modal model 444 component, where it is processed by the encoder component into latent space representation 124, as described throughout this application. It is then processed by latent space sensor fusion 445 components through the decoder component to output a multitude of space object state 222 data corresponding to the local volume of LEO space covered by sensors 110 on the independent sensing satellites 100.

[0084] The plurality of space object state 222 data corresponding to the local volume of LEO space covered by sensors 110 can be passed to Earth-based ground stations through satellite to ground wireless communications links.Versions with Software Embodied on a Centralized Earth-Based Device

[0085] It should be noted that, in some embodiments of the present invention, the invention can be implemented only as a software. That is, the invention is a method of processing and transforming data received from a combination of sensing satellites 100 and / or ground-based sensors without the need to implement additional physical components in satellites or on the ground that are not already available. In such cases, the invention is a method performed by processes embodied as instructions stored on a data storage device, preferably disposed on earth.

[0086] For example, referring to Fig. 11 , the invention is a method performed by software running on data processing hardware that is connected to a network of independent sensors 110 that can collect space object EM data 112 and signal of opportunity EM data 116 from the ground, air, and space. The collection of independent ground, air, or space based sensors 110 store their space object EM data 112, sensor state data 333, and sensor parameters data 334 in one or more data storage devices 114 which can be accessed by the proposed system components, such as the pre-processing component, through a variety of data link wireless and wired networks.Integration with Satellites 100

[0087] The various physical and processing components described of the invention herein are, to some extent, separable modules that can be physically separated depending on the application.

[0088] For example, as has been described but that warrants additional clarity, in some versions of the invention, one or more sensing satellites 100 are provided with a physical device, for example a data storage device, which comprises instructions to perform the processes described in this application. Such can be implemented by retrofitting such device onto sensing satellite 100 itself, preferably before such satellite is launched into orbit but also by whatever means current technologies permit (e.g., satellite servicing). Such can be provided by a system fully integrated with sensors 110, for example, a single device comprising all necessary components for the invention to be carried out, including sensors 110 and a data storage device with the instructions stored for processors to implement the methods described herein. In this way, the invention may be localized on a single satellite.

[0089] In other versions of the invention, a device may be integrated with, retrofitted to or otherwise in electrical / data communication with the existing control system and sensors of the satellite 100.

[0090] Devices of the present invention may, but need not, be disposed on a satellite capable of its own sensing; that is, it is possible that the invention is disposed on a satellite that does not perform sensing but rather receives data from other satellites with sensors 110.

[0091] Devices of the present invention may be distributed across a plurality of satellites or entirely disposed on a ground-based system 101 , as described. The “swarm” versions of the invention described herein do not require that each and every sensing satellite 100 of the plurality of satellites comprise a device embodied with the present invention. It is possible that only some sensing satellites 100 of the plurality of satellites have such a device, while the other sensing satellites 100 simply transmit their space object EM data 112 to a particular sensing satellite 100 or ground-based receiver that then is integrated with or in communication with a device that performs the methods of the invention described herein to generate latent space representation 124 and / or space object map 150.

[0092] The systems and methods described herein can be the sole purpose of a satellite or may be provided as an additional function on a satellite that has other purposes. The systems and methods described herein can be integrated into a unit that is incorporated into a satellite or can be the entire satellite itself. Such methods of construction depend on the particular use case and nothing in this application shall be interpreted to limit the method of integration of the present invention.

[0093] Any combination of the means of integration described in this application is possible.In other words, the “owner” of a system of the present invention need not own their own satellites to implement the present invention, or if they do, they can implement the present invention on some or all of such sensing satellites 100 that they own. Additionally, versions of the invention employing software embodied on a centralized Earth-based data storage device may be supplemented or accentuated with data (e.g., space object EM data 112 or latent space representation 124) received from sensing satellites 100 whether they own such satellites or not.Additional Variations

[0094] A number of additional variations on what is described above are contemplated.

[0095] For example, in another embodiment, sensing satellite 100 of the plurality of sensing satellites 100 is identified as the optimal sensing satellite to sense a particular region of space, with a focus on specific frequency bands, based on the sensing capabilities, position, orientation, and other attributes of such sensing satellite 100. This selection is performed using methods which are optimal with respect to the swarm and desired data by means such as compressed sensing projections. For example, the aforementioned map and signal of opportunity database can be employed as a prior estimate of the space objects in latent space representation 124 shared by the swarm satellites. Accordingly, methods including but not limited to discrete Fourier transform or wavelet graph methodscan be used to select sensing satellites 100 which are best suited to observe the estimated density and motion of space objects 30 in a desired area of observation.

[0096] In another embodiment, the present invention further comprises a system for opportunistic aperture synthesis and processing. With known signal characteristics, orbital parameters, and other data to estimate and predict the orientation, position, velocity, albedo, etc. of each sensing satellite 100 with respect to regions or volumes of space, and with communication present between same, where sensing satellite 100 are equipped with arrays of sensors 110 distributed along their surface, a given region of space can be opportunistically or ad hoc monitored by a virtual aperture formed by the a subset of satellites in the satellite swarm thus aligned to observe the region or volume of space. As such, there can be concurrent aperture synthesis occurring based on the relative configuration of sensing satellites 100 with respect to the region of space, and also with respect to the frequency band.

[0097] In another embodiment, the step of generating the State Data comprises determining the position(s) and velocity of Target Space Objects relative to any of the plurality of space-borne artificial satellites based on the difference in the travel time of when the electromagnetic signals reflected off of or emitted by the Target Space Object(s) were received and when the electromagnetic signals related to Emitting Space Objects or Reflecting Space Objects with Existing Signal Data were received.

[0098] In another embodiment, the Sensed Signal Data and the Ground-Based Sensor Data comprise data relate to electromagnetic signals in each of the radio, microwave, infrared, visible light, ultraviolet, and x-ray frequency bands; the step of storing the Sensed Signal Data, Ground-Based Sensor Data, and Existing Signal Data comprises storing the same in a first data storage device, the first data storage device comprising the instructions for generating the Latent Data; the step of generating the Latent Data comprises applying an encoder to the Sensed Signal Data, Ground-Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data relating each of the stored Sensed Signal Data, Ground-Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data one to the other; and the Position Data generated is a three-dimensional representation of the position of Target Space Object(s) and also represents the albedo, shape, and attitude of such Target Space Object(s).

[0099] In another embodiment: the Sensed Signal Data and the Ground-Based Sensor Data comprise data related to electromagnetic signals in each of the radio, microwave, infrared, visible light, ultraviolet, and x-ray frequency bands; the step of storing the Sensed Signal Data, Ground- Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data comprises storing the same in a first data storage device disposed on a first satellite of the plurality of space-borne artificial satellites, the first data storage device comprising instructions for generating the Latent Data; the stepof generating the Latent Data comprises applying an auto-encoder to the Sensed Signal Data; the step of storing the Latent Data comprises storing the Latent Data in the first data storage device and further comprises transmitting the Latent Data to either a second satellite of the plurality of space- borne artificial satellites or to a receiver on Earth; and the State Data generated is a three-dimensional representation of the relative position and velocity of Target Space Object(s) and also represents the albedo, shape, and attitude of such Target Space Object(s).

[0100] In another embodiment, a second satellite of the plurality of space-borne artificial satellites receives the Latent Data transmitted by the first satellite, and stores such Latent Data in a second data storage device disposed on the second satellite; and generates Position Data related to the Target Space Object(s) position using the Latent Data received from the first satellite as well as Latent Data generated from Sensed Signal Data obtained from the sensors of the second satellite.

[0101] Embodiments of the present invention are also directed to a system for detecting and mapping space objects, the system implementing the various methods described herein. For example, in one embodiment, such system comprises: a first computer device comprising at least one receiver and a first data storage device, the first data storage device comprising instructions that when executed: obtains data, using the at least one receiver, from a plurality of space-borne artificial satellites (“Sensed Signal Data”) and ground-based sensors (“Ground-Based Sensor Data”), each of the plurality of space-borne artificial satellites and ground-based sensors comprising at least one sensor receiving electromagnetic signals emitted directly by natural or artificial phenomena in space (“Emitting Space Objects”) or reflected off of other natural or artificial space objects (“Reflecting Space Objects”), the Sensed Signal Data of any given satellite of the plurality of space-borne artificial satellites and the Ground-Based Sensor Data of any given sensor of the plurality of ground-based sensors comprising information relating to electromagnetic signals reflected off of or emitted by at least one target space object (“Target Space Object(s)”), the Sensed Signal Data and Ground-Based Sensor Data comprising data relating to a plurality of distinct frequency bands of the electromagnetic signals reflected off of or emitted by the Target Space Object(s); obtains data (“Sensed Signal Time Data”), using the at least one receiver, from each of the plurality of space-borne artificial satellites and ground-based sensors relating to the time such Sensed Signal Data or Ground-Based Sensor Data was received by the given satellite or ground basedsensor of the plurality of space-borne artificial satellites or ground-based sensors; obtains data, using the at least one receiver, relating to the positions of known or determinable sources or reflections of artificial or natural electromagnetic signals (“Existing Signal Data”); stores the Sensed Signal Data, Ground-Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data in the first data storage device;generates a compressed latent data representation (“Latent Data”) of the stored Sensed Signal Data, Ground-Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data; stores the Latent Data; and generates position data (“Position Data”) related to the Target Space Object(s)’s position using the Latent Data.

[0102] In the system described in the foregoing paragraph, the first computer device may be disposed on a first satellite of the plurality of space-borne artificial satellites and the system also comprises a transmitter capable of transmitting the Latent Data. The system of may also comprise a second computer device comprising at least one receiver and a second data storage device disposed on a second satellite, the second data storage device comprising instructions that when executed: obtains the Latent Data, using the at least one receiver, from the first satellite; and generates Position Data related to the Target Space Object(s) position using the Latent Data received from the first satellite as well as Latent Data generated from Sensed Signal Data obtained from a second satellite of the plurality of space-borne artificial satellites. The system may also be such that the Sensed Signal Data and the Ground-Based Sensor Data comprise data relating to electromagnetic signals in each of the radio, microwave, infrared, visible light, ultraviolet, and x-ray frequency bands; the first data storage device further comprises instructions to generate the Latent Data by applying an autoencoder to the Sensed Signal Data, Ground-Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data relating each of the stored Sensed Signal Data, Ground-Based Sensor Data, Sensed Signal Time Data, and Existing Signal Data one to the other or to other data that creates a relationship between such data; and the Position Data generated is a three-dimensional representation of the position of Target Space Object(s) and also represents the albedo, shape, and attitude of such Target Space Object(s).Further Definitional Issues and Statements of Scope

[0103] Note that in the specification and claims, “about” or “approximately” means within twenty percent (20%) of the numerical amount cited. All computer software disclosed herein may be embodied on any non-transitory computer-readable medium (including combinations of mediums), including without limitation CD-ROMs, DVD-ROMs, hard drives (local or network storage device), USB keys, other removable drives, ROM, and firmware.

[0104] It is noted that whenever processors are described as performing an action in this application, or some process is being performed, it may be presumed to be processors executing software instructions 120 that are stored in a data storage device, for example data storage device 114, unless the context requires otherwise.

[0105] The terms "about" or "approximately" as used herein, mean an acceptable error for an articular recited value, which depends in part on how the value is measured or determined. In certain embodiments, "about" can mean one or more standard deviations. When the antecedent term "about" is applied to a recited range or value it denotes an approximation within the deviation in the range or value known or expected in the art from the measurement method. For removal of doubt, it should be understood that any range stated in this written description that does not specifically recite the term "about" before the range or before any value within the stated range inherently includes such term to encompass the approximation within the deviation noted above.

[0106] Embodiments of the present invention can include every combination of features that are disclosed herein independently from each other. Although the invention has been described in detail with particular reference to the disclosed embodiments, other embodiments can achieve the same results. Variations and modifications of the present invention will be obvious to those skilled in the art and it is intended to cover in the appended claims all such modifications and equivalents. The entire disclosures of all references, applications, patents, and publications cited above are hereby incorporated by reference. Unless specifically stated as being “essential” above, none of the various components or the interrelationship thereof are essential to the operation of the invention. Rather, desirable results can be achieved by substituting various components and / or reconfiguration of their relationships with one another. The terms, “a”, “an”, “the”, and “said” mean “one or more” unless context explicitly dictates otherwise.

[0107] The preceding examples can be repeated with similar success by substituting the generically or specifically described components and / or operating conditions of embodiments of the present invention for those used in the preceding examples.

[0108] Optionally, embodiments of the present invention comprise software, that is, executable instructions stored on a storage device on sensing satellite 100 (for example, data storage device 114), that when executed by one or more processors or computers (the terms “processor” or “computer” being interchangeably used herein) on sensing satellite 100, performs the processes described herein. Embodiments of the present invention can include a general or specific purpose computer or distributed system programmed with computer software implementing steps described above, which computer software may be in any appropriate computer language, including but not limited to C++, FORTRAN, BASIC, Java, Python, Linux, assembly language, microcode, distributed programming languages, etc. The apparatus may also include a plurality of such computers / distributed systems (e.g., connected over terrestrial and non-terrestrial communication networks, the Internet and / or, one or more intranets) in a variety of hardware implementations. For example, data processing can be performed by an appropriately programmed microprocessor, computing cloud, software-defined radio, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or the like, in conjunction with appropriate memory, network, and bus elements. One ormore processors and / or microcontrollers can operate via instructions of the computer code and the software can be stored on one or more tangible non-transitive memory storage devices.

Claims

CLAIMSWhat is claimed is:

1. A method of mapping space objects, the method comprising: obtaining data from a plurality of space-borne artificial satellites (“Sensed Signal Data”) and ground-based sensors (“Ground-Based Sensor Data”), each of the plurality of space-borne artificial satellites and ground-based sensors comprising at least one sensor receiving electromagnetic signals emitted directly by natural or artificial phenomena in space (“Emitting Space Objects”) or reflected off of other natural or artificial space objects (“Reflecting Space Objects”), the Sensed Signal Data of any given satellite of the plurality of space-borne artificial satellites and the Ground-Based Sensor Data of any given sensor of the plurality of ground-based sensors comprising information relating to electromagnetic signals reflected off of or emitted by at least one target space object (“Target Space Object(s)”), the Sensed Signal Data and the Ground-Based Sensor Data comprising data relating to a plurality of distinct frequency bands of the electromagnetic signals reflected off of or emitted by the Target Space Object(s); obtaining data relating to the positions of known or determinable sources or reflections of artificial or natural electromagnetic signals (“Existing Signal Data”); storing the Sensed Signal Data, Ground-Based Sensor Data, and Existing Signal Data; processing the Sensed Signal Data, Ground-Based Sensor Data, and Existing Signal Data to generate rough, partial, or ambiguous estimates of the position, velocity, attitude, novel views, and other data of Target Space Object(s) (“Candidate State Data”); storing the Candidate State Data; and generating a compressed latent data representation (“Latent Data”) of the stored Sensed Signal Data, Ground-Based Sensor Data, Candidate State Data, and Existing Signal Data; storing the Latent Data; and fusing the Latent Data into a latent data representation (“Fused Latent Data”); generating position, velocity, attitude, novel views, and other data (“State Data”) related to the Target Space Object(s)’s state using the Fused Latent Data.

2. The method of claim 1 , wherein the Sensed Signal Data and the Ground-Based Sensor Data comprises data relating to electromagnetic signals in each of the radio, microwave, infrared, visible light, ultraviolet, and x-ray frequency bands.

3. The method of claim 1 , wherein the step of processing the Sensed Signal Data, Ground- Based Sensor Data and Existing Signal Data comprises: obtaining data related to the state and parameters of the sensors of the space-borne artificial satellites and ground-based sensors (“Sensor State Data”); andemploying a plurality of analytic algorithms determined by the format and underlying frequency of the Sensed Signal Data and Ground-Based Sensor Data to generate the Candidate State Data.

4. The method of claim 3, wherein the Candidate State Data comprises two-line element data.

5. The method of claim 3, wherein the sensors of the space-borne artificial satellites and ground- based sensors comprise optical sensors, and the plurality of analytic algorithms comprises a multiview stereo computer vision algorithm applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relate to optical wavelengths, to generate the Candidate State Data, wherein the Candidate State Data represents the position and velocity of the target space objects relative to the sensors of the space-borne artificial satellites and ground-based sensors.

6. The method of claim 3, wherein the sensors of the space-borne artificial satellites and ground- based sensors comprise optical sensors, and the plurality of analytic algorithms comprises a multiview synthesis algorithm applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relate to optical wavelengths, to convert the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data from two-dimensional space into a three-dimensional representation to be used in the step of generating Candidate State Data.

7. The method of claim 3, wherein the sensors of the space-borne artificial satellites and ground- based sensors comprise radio frequency sensors, and the plurality of analytic algorithms comprises aperture synthesis algorithms applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relates to radio frequency data, to generate the Candidate State Data, wherein the Candidate State Data represents the position and velocity of space objects in the data relative to the sensors.

8. The method of claim 3, wherein the sensors of the space-borne artificial satellites and ground- based sensors comprise radio wavelength phased arrays, and the plurality of analytic algorithms comprises beamforming algorithms to spatiotemporally optimize the received signal processing to maximize the phased array antenna's sensitivity to signals arriving from a specific direction while simultaneously suppressing interfering signals and noise arriving from other directions, the algorithms applied to the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data that relates to data received from the radio wavelength phased arrays sensors, to generate the Candidate State Data.

9. The method of claim 3, wherein the sensors of the space-borne artificial satellites and ground- based sensors comprise radio frequency sensors, and the plurality of analytic algorithms comprises range-based, time-based, and angle-based relative localization algorithms applied to the SensedSignal Data, Ground-Based Sensor Data and Existing Signal Data that comprises radio frequency data.

10. The method of claim 3, wherein the Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data further comprises data that is known and data that is unknown, and the step of processing such data further comprises: regarding data that is not previously known, amplifying and storing the data in a database (“Known Signal Database”); and regarding data that is known, checking such known data against data in the Known Signal Database and repeating the same for a period of time T.11 . The method of claim 10, wherein the step of processing the Sensed Signal Data, Ground- Based Sensor Data and Existing Signal Data further comprises applying a bi-static range processing algorithm and a time difference of arrival (“TDOA”) algorithm, using data obtained from the Known Signal Database.

12. The method of claim 11 , wherein the step of processing the Sensed Signal Data, Ground- Based Sensor Data and Existing Signal Data further comprises establishing a bi-static range ellipsoid capturing a range of possible positions of the Target Space Object and storing the same as Candidate State Data.

13. The method of claim 12, wherein the Candidate State Data comprises hyperboloids generated by the TDOA algorithm, the hyperboloids capturing a range of possible positions of the Target Space Object.

14. The method of claim 13, further comprising combining a plurality of Candidate State Data and storing as Candidate State Data the intersection of the hyperboloids generated by the TDOA algorithm, thereby reducing the ambiguity of the resulting Candidate State Data.

15. The method of claim 1 , further comprising: obtaining time-related data (“Sensed Signal Time Data”) from each of the plurality of space- borne artificial satellites and ground-based sensors relating to the time such Sensed Signal Data or Ground-Based Sensor Data was received by the given satellite or ground based-sensor of the plurality of space-borne artificial satellites or ground-based sensors; and storing the Sensed Signal Time Data; and wherein the steps of generating Candidate State Data, generating Latent Data and fusing the Latent Data further comprises using the Sensed Signal Time Data.

16. The method of claim 1 , wherein the step of generating Latent Data comprises applying a first encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the radio frequency band, a second encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the microwave frequency band, a third encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the infrared frequency band, a fourth encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the visible light frequency band, a fifth encoder to the Sensed Signal Data and the Ground- Based Sensor Data in the ultraviolet frequency band, and a sixth encoder to the Sensed Signal Data and the Ground-Based Sensor Data in the x-ray frequency band, wherein each encoder comprises neural network architectures.

17. The method of claim 1 , wherein the step of generating Latent Data comprises applying a single encoder to the Sensed Signal Data and the Ground-Based Sensor Data, wherein the single encoder comprises a neural network architecture.

18. The method of claim 16, wherein the step of generating Fused Latent Data further comprises: generating distinct data within the Latent Data for a particular moment in time of the SensedSignal Data, Ground-Based Sensor Data and Existing Signal Data; and applying at least one process selected of the group consisting of: concatenation, pooling, weighted mean, median, and other combination processing.

19. The method of claim 17, wherein the step of generating Fused Latent Data further comprises processing the Latent Data by attention processes selected from the group consisting of: selfattention, cross-attention, dot-product attention, and multi-head attention processes, to learn contextual parameters that optimally combine the Latent Data.

20. The method of claim 17, wherein the step of generating Fused Latent Data further comprises using Bayesian estimation processing in the case that Latent Data relates to Sensed Signal Data, Ground-Based Sensor Data and Existing Signal Data received over a period of time.21 . The method of claim 20, wherein the Bayesian estimation processing comprises: using the Latent and Fused Data related to a first moment in time to predict Fused Latent Data (“Predicted Fused Latent Data”) related to a second moment in time.

22. The method of claim 21 , wherein the Bayesian estimation processing comprises: relating the Predicted Fused Latent Data of the second moment in time to the Fused Latent Data of the first moment in time using a recursive fusion filtering process selected from the group consisting of: Kalman filtering, Kalman net and particle filers.

23. The method of claim 21 , wherein the step of using the Latent and Fused Data related to the first moment in time to the Predicted Fused Latent Data related to the second moment in time further comprises: modeling the Predicted Fused Latent Data as a hidden state of stored data within a recurrent neural network, the model evolving based on inputted Latent and Fused Latent Data related to the first moment; and modelling within the recurrent neural network the covariance of error between the Predicted Fused Latent Data and an actual Fused Latent Data based on a plurality of Latent Data and Predicted Fused Latent Data; projecting the hidden state to a Kalman gain; and updating the current time Fused Latent Data to be the sum of the Predicted Fused Latent Data and the Kalman gain multiplied by the difference between the Predicted Fused Latent Data and the actual Latent Data.

24. The method of claim 23, wherein the recurrent neural network is a reservoir computer.

25. The method of claim 1 , wherein the step of generating State Data comprises applying a decoder or plurality of decoders (“decoder(s)”), wherein each decoder comprises a neural network architecture or plurality of neural network architectures.

26. The method of claim 25, wherein the decoders generate visual representations of the Fused Latent Data in the form of a range-doppler map.

27. The method of claim 25, wherein the decoders generate visual representations of the Fused Latent Data in the form of fixed frames of reference to the earth.

28. The method of claim 1 , wherein the step of storing the Sensed Signal Data, Ground-Based Sensor Data, Candidate State Data, and Existing Signal Data comprises storing the same in a first data storage device disposed on a first satellite of the plurality of space-borne artificial satellites, the first data storage device comprising the instructions for generating the Latent Data.

29. The method of claim 1 , wherein the step of generating the State Data is performed by instructions stored in a data storage device disposed on Earth.

30. The method of claim 1 , wherein the step of storing the Sensed Signal Data, Ground-Based Sensor Data, Candidate State Data, and Existing Signal Data comprises storing the same in a first data storage device disposed on Earth, the first data storage device comprising the instructions for generating the Latent Data.31 . The method of claim 1 , wherein the State Data also represents the albedo, shape, or attitude of the Target Space Object(s).

32. The method of claim 1 , wherein the State Data generated is a three-dimensional representation of the relative position and velocity of Target Space Object(s) within a certain distance of a satellite of the plurality of space-borne artificial satellites.

33. The method of claim 1 , further comprising a first satellite of the plurality of space-borne artificial satellites transmitting its Sensed Signal Data or Latent Data to a second satellite of the plurality of space-borne artificial satellites.

Images