Edge cloud image orchestration of spaceborne data

Abstract

The disclosed technology generally relates to edge cloud image orchestration of spaceborne data. In one example of the technique, reference embedding is provided. A downlink session with the orbital satellite is established. During a downlink session: a spacecraft embedding is received from an orbiting satellite. The spacecraft embedding is generated by applying an embedding generation model to corresponding image portions that are stored on an orbiting satellite and obtained from sensors on the orbiting satellite. In vector space, spacecraft embedding is compared to reference embedding. It is determined which portions of the image are high-valued portions of the image based at least in part on the comparison. The orbital satellite is communicated to which portions of the image in the portions of the image are high-valued portions of the image. A high value portion of an image is received from an orbital satellite.

Description

Background Art

[0001] Spaceborne data, such as satellite imagery, has many useful applications in agriculture, biodiversity, mapping, conservation, disaster response, education, fisheries, forestry, geology, landscape analysis, meteorology, oceanography, and regional planning. For example, spaceborne data can be used to monitor hazards such as fires, volcanic activity, landslides, and avalanches. It can also be used for rapid post-disaster mapping. It can be used to locate ships in the ocean or aircraft on the tarmac. It can be used for various hydrological applications, such as drought and soil moisture monitoring and discovering hidden river channels. These are just a few of the many possible applications for spaceborne data. Summary of the Invention

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] Briefly, the disclosed technology generally relates to edge cloud image orchestration of onboard data. A plurality of reference embeddings are provided. A downlink session is established with an orbiting satellite. During the downlink session: a plurality of spacecraft embeddings are received from the orbiting satellite. Spacecraft embeddings in the plurality of spacecraft embeddings are generated on the orbiting satellite by applying an embedding generation model to corresponding image portions, which are stored on the orbiting satellite and obtained from sensors on the orbiting satellite. The spacecraft embeddings in the plurality of spacecraft embeddings are compared in vector space with reference embeddings in a plurality of reference embeddings. Which portions of the image are high-value portions of the image are determined based at least in part on the comparison. Which portions of the image in the image portions are high-value portions of the image are communicated to the orbiting satellite. The high-value portions of the image are received from the orbiting satellite.

[0004] Other aspects and applications of the disclosed technology will be understood upon reading and understanding the accompanying drawings and description. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting and non-comprehensive examples of the present disclosure are described with reference to the following drawings. In the drawings, unless otherwise specified, like reference numerals refer to like parts throughout the various figures. These drawings are not necessarily drawn to scale.

[0006] For a better understanding of the present disclosure, reference will be made to the following detailed description which is to be read in conjunction with the accompanying drawings, in which:

[0007] Figure 1 is a block diagram illustrating an example of a network connection system;

[0008] Figure 2 is a block diagram illustrating an example of a system for edge-cloud image orchestration of spaceborne data;

[0009] Figure 3 is a flow chart illustrating an example process for edge cloud image orchestration for spaceborne data;

[0010] Figure 4 is a block diagram illustrating one example of a suitable environment in which various aspects of the present technology may be employed; and

[0011] Figure 5 is a block diagram illustrating one example of a suitable computing device in accordance with various aspects of the disclosed technology. DETAILED DESCRIPTION

[0012] Constrained environment devices can transmit information within limited time intervals and may have limited computing resources. For example, onboard data is transmitted to the Earth by performing downlink transmissions within a scheduled time, where a downlink session may last about ten minutes because the satellite passes by a ground station. In addition, the resources on the orbiting satellite (such as computing resources) are relatively limited. Therefore, a constrained environment device may contain more data than can be fully analyzed on the constrained environment device within a reasonable time period. For example, an orbiting satellite contains more data than can be fully analyzed on the satellite within a reasonable time period. Additionally, the amount of data that can be downlinked to the Earth is limited, so the onboard data on each satellite is significantly greater than the amount of onboard data that can be downlinked within a reasonable time period. This may result in valuable data being lost, and this may prevent the sensors on the constrained environment device and the data captured by the sensors from being fully utilized.

[0013] Therefore, in some examples, during each downlink session, edge-cloud image orchestration is performed to determine which image tiles stored on the constrained environment device have high values, and during the downlink session, the image tiles determined to have the highest values ​​are delivered from the constrained environment device to a receiving entity, such as a ground station and one or more connected data centers. The constrained environment device divides the image captured and stored on the constrained environment device into image tiles of uniform size. An embedding generation model is used to generate embeddings from the image tiles, and the embeddings are stored on the constrained environment device.

[0014] During the downlink session, the embedding is transmitted to the receiving entity. Next, during the downlink session, the receiving entity compares the downlinked embedding with a reference embedding stored at the receiving entity. These comparisons, along with various other suitable information, are used to determine which image tiles have the highest values. The receiving entity then requests the image tiles determined to have the highest values, and these image tiles are transmitted from the constrained environment device to the receiving entity during the downlink session.

[0015] Descriptive System

[0016] Figure 1 is a block diagram illustrating an example of a system. Figure 1 and in the instructions Figure 1 The corresponding description illustrates an example system for illustrative purposes, which does not limit the scope of the present disclosure. According to some examples, system 100 is described as follows. System 100 includes network 130 and client device 151, client device 152, service device 161, service device 162, constrained environment device 171, and constrained environment device 172, all connected to network 130. In some examples, system 100 operates as follows.

[0017] Each of the client device 151, the client device 152, the service device 161, the service device 162, the constrained environment device 171, and the constrained environment device 172 includes Figure 5 1. Service device 161 and service device 162 are part of one or more distributed systems.

[0018] Service device 161 and service device 162 are each part of the service provided on behalf of the client. Each client can communicate with the service via one or more devices (such as client device 151 and client device 152). The service includes various aspects associated with data collected by constrained environment devices (including constrained environment device 171 and constrained environment device 172). Each constrained environment device in the constrained environment device is a device in a constrained environment. Each constrained device in the constrained device can be, for example, a satellite in Earth orbit, a device in deep space (such as a probe or spacecraft in deep space), a drone, a device on a fixed platform (such as an oil platform) (such as a fixed tower or a fixed sensor), an Internet of Things (IoT) device in a constrained environment, etc.

[0019] For example, the service may include various aspects related to collecting data from constrained environment devices, analyzing data from constrained devices, searching data from constrained devices, and the like.

[0020] Some of the constrained environment devices (e.g., 171 and 172) operate as follows: The constrained environment device generates and stores images using sensors. The constrained environment device generates embeddings from the images or portions of the images using an embedding generation model.

[0021] In some examples, a service comprising service device 161 and service device 162 operates as follows. The service is configured to receive information from constrained environment devices. Each constrained environment device communicates with the service during a separate downlink session. During the downlink session from the constrained environment device to the service, embeddings are passed from the constrained environment device to the service. Based on the embeddings, the service determines which images or image portions have the highest values. The service then requests those images or image portions from the constrained environment devices determined to have the highest values. The constrained environment devices then send the requested images or image portions to the service during the downlink session.

[0022] In some examples, the downlink session is a session between an orbiting satellite and a ground station. In other examples, the downlink session involves two or more orbiting satellites, such as via satellite crosslinks, inter-satellite optical laser links, or other suitable forms of satellite communication.

[0023] The network 130 may include one or more computer networks, including wired and / or wireless networks, each of which may be, for example, a wireless network, a local area network (LAN), a wide area network (WAN), and / or a global network such as the Internet. On a collection of interconnected LANs, including LANs based on different architectures and protocols, routers act as links between LANs, enabling messages to be sent from one LAN to another. Furthermore, communication links within a LAN typically include twisted pair or coaxial cables, while communication links between networks may utilize analog telephone lines, dedicated digital lines including all or part of T1, T2, T3, and T4, integrated services digital networks (ISDN), digital subscriber lines (DSL), wireless links including satellite links, and / or other communication links known to those skilled in the art. For example, such satellite links may include satellite interconnect communications and various types of satellite downlinks, including satellite downlinks in the X, S, Ka, and ultra-high frequency (UHF) bands. Furthermore, remote computers and other related electronic devices may be remotely connected to a LAN or WAN via a modem and a temporary telephone link. Network 130 may include various other networks, such as one or more networks using local network protocols (such as 6LoWPAN, ZigBee, etc.). Essentially, network 130 may include any suitable network-based communication method by which information can be transmitted between client device 151, client device 152, service device 161, service device 162, constrained environment device 171, and constrained environment device 172. Although each device is shown as being connected to network 130, this does not necessarily mean that each device communicates with every other device shown. In some examples, some of the devices shown communicate only with some of the other devices / services shown via one or more intermediary devices. Furthermore, although network 130 is illustrated as a single network, in some examples, network 130 may instead include multiple networks that may or may not be connected to each other, with some of the devices shown communicating with each other via one of the multiple networks, while other of the devices shown communicate with each other using a different network from the multiple networks. Furthermore, some of the links (such as satellite links) may be limited or intermittent.

[0024] System 100 may include Figure 1 More or fewer devices are shown, where Figure 1 Shown as an example only.

[0025] Figure 2 is a block diagram illustrating an example system. System 200 may be Figure 2

[0045] An example of system 100 is provided. According to some examples, system 200 is described as follows. System 200 includes client device 251, client device 252, service system 260, satellite device 271, and satellite device 272. Service system 260 includes one or more distributed systems. In some examples, system 200 operates as follows.

[0026] Satellite devices (including satellite devices 271 and 272) are devices on satellites orbiting the Earth. Satellite devices store onboard data collected by orbiting satellites. Onboard data includes images captured by satellites. Service system 260 provides services on behalf of clients, where the services include at least one service associated with the onboard data collected by the satellite devices. Given that the satellites orbit the Earth, service system 260 is located on Earth. In addition, service system 260 includes a ground station and one or more data centers connected to the ground station. Each client communicates with service system 260 via one or more devices (such as client device 251 and client device 252).

[0027] The resources onboard orbiting satellites, such as computing resources, are relatively limited compared to the resources of the service system 260. Consequently, orbiting satellites contain more data than can be fully analyzed onboard the satellite within a reasonable timeframe. Additionally, the amount of data that can be downlinked to Earth is limited, so the amount of onboard data on each satellite is significantly greater than the amount of onboard data that can be downlinked within a reasonable timeframe. This can result in valuable data being lost and prevent the sensors onboard the satellites and the data they capture from being fully utilized. However, various aspects of the present disclosure enable the most useful analysis to be performed onboard the satellites, and enable the highest value images on the satellites to be downlinked first.

[0028] In some examples, some satellite devices, such as satellite devices 271 and 272, operate as follows.

[0029] Satellite equipment includes sensors and also includes devices for storing data. Sensors on orbiting satellites collect data, including raw images and image metadata, which are all stored on the satellite. Examples of image metadata include time, slope, azimuth, altitude, scene center latitude, scene center longitude, viewing angle, angle of incidence, solar altitude, solar azimuth, or other suitable metadata. Sensors can include a variety of different sensor types, including cameras, synthetic aperture radars, thermal imaging sensors, hyperspectral sensors, video sensors, or other suitable sensor types.

[0030] The images provided by the sensor can include two-dimensional images, three-dimensional representations, and images such as synthetic aperture radar images. Each type of image discussed above is a type of onboard data that can be captured by satellite equipment. The satellite equipment divides each of the stored images into smaller tiles. For example, in one example, a full-size image of 8,192 pixels by 8,192 pixels can be broken down into 256 evenly sized tiles of 512 pixels by 512 pixels. Smaller tiles are easier to process, analyze, and downlink.

[0031] Then, one or more of the satellite devices on the satellite (e.g., satellite device 271 or 272) converts each tile in the image into an embedding. The tile is retained and continues to be stored in the satellite device, and the embedding is also stored in the satellite device. The embedding is a feature vector of floating point numbers, which is an ordered list of numeric vectors such that for two similar images, the corresponding feature vectors are close to each other in the vector space. Similarly, for two different images, the feature vectors are not close to each other in the vector space. Embedding is sometimes also called a "representation."

[0032] For example, in some examples, an unsupervised machine learning embedding generation model is used to convert each tile in the image into an embedding. In one example, a simple Siamese model trained on a large number of images (such as over 100,000 different image tiles) is used to convert each tile in the image into a 512-dimensional feature vector. Various suitable numbers of dimensions can be used in various examples, such as at least 256 dimensions in some examples. The unsupervised machine learning embedding generation model can also be referred to as an unsupervised representation learning model.

[0033] In other examples, other suitable methods are used to generate embeddings. For example, in some examples, self-supervised representation learning techniques or supervised representation learning methods are used to generate embeddings. The embedding of a tile can be significantly smaller than the corresponding tile, such as 1 / 12 the size of the corresponding tile in one example. Once the satellite captures the image, the image is divided into tiles, and the tiles are converted into embeddings.

[0034] In some examples, the satellite has dozens or hundreds of different models on board that can be used to run against any of the image tiles. These models can include, for example, artificial intelligence (AI) models that are airplane detector models, ship detector models, building detector models, cloud detector models, methane detector models, fire detector models, car detector models, and the like. These models can include, for example, classification models, object detection models, and segmentation models. For various reasons, it may be desirable to run a specific model. For example, it may be desirable to use a cloud detection model to exclude or deprioritize tiles that do not contain useful information due to the presence of clouds in the tile.

[0035] Running models that detect specific objects or phenomena may be useful for reasons of prioritization or identifying data that may require immediate attention. For example, if an image tile includes a potentially dangerous hazard, such as a forest fire, it may be desirable to downlink the image tile as quickly as possible, rather than waiting for the next scheduled downlink. As another example, it may be determined that a ship detector model should not be run on tiles where the image is occupied by land or obscured by clouds, fog, or smoke, thereby saving resources for other operations on the satellite. However, the satellite does not have sufficient power and resources to run all models for all image tiles. Therefore, optionally, in some examples, model orchestration is performed in the satellite. In other examples, model orchestration is not performed in the satellite. An example of a system 200 that performs model orchestration operates as follows.

[0036] In model orchestration performed on the satellite, one or more of the satellite devices (e.g., satellite device 271 or 272) uses the embeddings to determine which models should be run on which tiles. After determining which models should be run on which tiles, the satellite devices (e.g., satellite devices 271 and 272) run the determined models on the tiles. The satellite devices then take appropriate actions based on the results of the models run on the tiles.

[0037] For example, if the model indicates that an image tile may indicate the presence of a significant hazard, the satellite equipment can cause the image tile to be downlinked as quickly as possible. In some examples, a short message or other type of alert can be downlinked as quickly as possible, rather than immediately downloading the image tile, as a short message may be more easily downlinked than an image. Model orchestration on the satellite can more efficiently use onboard resources and enable more models to run with the same resources.

[0038] Optionally, in some examples, satellite devices (e.g., satellite devices 271 and 272) perform downlink prioritization on the satellite using embeddings. In examples where model orchestration and downlink prioritization are performed on the satellite, downlink prioritization occurs after model orchestration is performed on the satellite. The downlink prioritization performed on the satellite utilizes a target embedding set. The target embedding set used for downlink prioritization is generated by service system 260.

[0039] The embeddings included in the target embedding set are determined based on the client's priorities. The target embedding set is uploaded to the satellite device from the service system 260 and stored on the satellite device. The satellite device automatically classifies which tiles should be transmitted by comparing the embeddings with the target embedding set. Comparing the embeddings with the target embedding set includes determining which embeddings are close in vector space to the embeddings in the embedding set.

[0040] As discussed above, the embeddings included in the embedding set are determined based on the client's priorities. For example, if the client wants to prioritize downlinking of cloud-free imagery, downlink prioritization can be used to infer which image tiles appear most cloud-free. Onboard downlink prioritization and onboard model orchestration are determined on a tile-by-tile basis based on each embedding in the embedding set from which it was generated. Downlink prioritization performed onboard the satellite saves time and bandwidth, ensures access to relevant assets, and prioritizes decision making.

[0041] In some examples, some information from other tiles in the image (such as average pixel values) may also be used in the determinations made for each tile in the tile. In some examples, onboard downlink prioritization uses a rule-based decision tree to perform downlink prioritization. In some examples, the decision tree uses embedded and image information to perform onboard downlink prioritization. The image information used during downlink prioritization may include image metadata. In some examples, the rules are manually defined by the client. In some examples, the rules are defined based on an analysis of historical usage patterns. In some examples, the historical usage patterns include various suitable information, such as existing client requests, existing client searches, historical visits by the client, sales patterns with the client, or other suitable information. In some examples, the image data used includes geographic information. For example, some clients may have specific geographic areas of interest, and prioritization may be based in part on whether the tile is located in the geographic area of ​​interest to the client.

[0042] Each of the satellites performs downlink transmissions at periodic time intervals, wherein the satellites communicate with the service system 260. As discussed above, the service system 260 provides services on behalf of the client, wherein the services include at least one service associated with onboard data collected by the satellite devices (e.g., satellite devices 271 and 272). For example, the services may include storing onboard data downlinked from the satellite devices, analyzing onboard information from the constrained devices, searching for onboard information, etc. During a separate downlink session between the satellite devices and the service system 260, each of the satellite devices transmits information to the service system 260. During the downlink session, some image tiles on the satellite devices are downlinked to the service system 260. In the example of using onboard downlink prioritization, the downlink transmission of the image tiles is classified at least in part based on the onboard downlink prioritization discussed above.

[0043] Serving system 260 stores an embedding database. The embedding database is a database that stores a relatively large number of reference embeddings that can be used to assist in determining which image tiles have the highest values. The values ​​of the image tiles are based at least in part on the client's priorities. When a downlink session occurs between a satellite and serving system 260, embeddings are sent to serving system 260 from satellite devices on the satellite (e.g., satellite devices 271 and 272). For example, in some examples, the satellite devices send embeddings for each image captured by the satellite since the last downlink transmission. In other examples, some embeddings are excluded based on downlink priority sorting performed on the satellite. In either case, serving system 260 then compares the received embeddings with the reference embeddings in the embedding database. This comparison determines which received embeddings are closest to the reference embeddings in the vector space.

[0044] Serving system 260 then analyzes the received embeddings that are determined to be closest to the reference embedding to determine which of these embeddings correspond to high-value image tiles. As discussed above, the values ​​of the image tiles are based at least in part on the client's priority. Various suitable information (such as existing client requests, existing client searches, historical visits by the client, sales patterns with the client, or other suitable information) can be combined to determine which image tiles have the highest values ​​based on the similarity of the determined embeddings to the reference embeddings. After identifying the highest-value image tiles, serving system 260 transmits the highest-valued image tiles to the satellite device for downlink transmission.

[0045] Next, the satellite device downlinks the indicated high-value tiles. The service system 260 then receives the downlinked tiles and subsequently stores the received tiles. In some examples, the determination and downlinking of high-value image tiles occurs within a few seconds during a single downlink session and occurs during each scheduled downlink session. In some examples, the entire downlink session is approximately ten minutes when the satellite passes by the ground station. The determination of high-value image tiles during the downlink session between the satellite and the service system 260 on the ground allows for relatively seamless communication between the satellite and the service system 260, increases the speed of decision-making, and ensures the capture of high-value data. After the high-value tiles are downlinked, the service system 260 receives the downlinked tiles and subsequently stores the received tiles.

[0046] As discussed above, in some examples, the downlink session is between an orbiting satellite and service system 260. In other examples, the downlink session involves two or more orbiting satellites, such as via satellite crosslinks, inter-satellite optical laser links, or other suitable forms of satellite communication.

[0047] In some examples, onboard downlink prioritization is not performed, but edge-cloud image orchestration is performed. In some examples, onboard downlink prioritization and edge-cloud image orchestration are performed. In examples where onboard downlink prioritization and edge-cloud image orchestration are performed, the specific image tiles downlinked during the downlink session are based on the onboard downlink prioritization and edge-cloud image orchestration.

[0048] The images collected from the various downlink sessions are stored in the serving system 260 and have associated metadata stored according to the Spatiotemporal Asset Catalog (STAC) specification. In some examples, the associated metadata is stored in a separate database. The serving system 260 enables clients to perform reverse image searches based on the embeddings of the images. In some examples, the reverse image search comprises querying the metadata database of the STAC project via an application programming interface (API) or a user interface. In this way, a user can perform a visual search of existing images in the database to find similar images of interest. The embeddings are also used to recommend similar images to the user. For example, if the client is an analyst interested in mapping the locations of small solar installations within the United States, the serving system 260 can recommend additional images where the solar installations may be visible.

[0049] The serving system 260 enables clients to run models against any satellite imagery stored in the serving system 260. Some examples of the serving system 260 have a large number of models, such as hundreds of models, that can be run against an image. These models may include, for example, artificial intelligence (AI) models that are airplane detector models, ship detector models, building detector models, cloud detector models, and so on. However, running each model against all images would be relatively inefficient. Model orchestration is performed by the serving system 260. Therefore, in some examples, cloud model orchestration is optionally performed. One example of the system 200 that performs the optional cloud model orchestration operates as follows.

[0050] Model orchestration uses embeddings of imagery taken from satellites to determine which models should be run on which images. For example, a model trained to find airplanes on the tarmac may work one day, but fail the next day if it's snowing at a given airport of interest. Model orchestration, performed in serving system 260, alerts the client to this change and prevents the model from providing incorrect results to the user. Using the embeddings, a determination is made for each model and each image as to whether the model should be run on that image. As another example, a model that detects poultry farms might only be run on tiles corresponding to rural or semi-rural locations that are not water areas and have structures on them. In this way, a model that detects poultry farms can avoid running on tiles where it is not necessary to run the model. Model orchestration, performed in serving system 260, enables more efficient use of serving system 260's resources.

[0051] The example of system 200 may be used for a variety of different example use cases for a variety of different example clients.

[0052] For example, an example client is an oil and gas company that utilizes an example of system 200 to reduce the time required for the client to detect and resolve a large-scale methane leak at its wells, pipelines, and refineries spread across an oil field covering hundreds of square miles. Rapidly detecting methane leaks can significantly reduce the client's compliance, regulatory, and enforcement risks. In this example, a methane detection model is deployed on a satellite constellation to enable rapid notification of methane leaks. Each time a satellite in the constellation passes over an oil field of interest, the model is "turned on" within the satellite to search for methane leaks. In this example, this is accomplished on the satellite as follows.

[0053] Of all the images of oil fields captured by satellites, approximately 99% of the images or image tiles will be cloudy or contain no methane leaks. As part of the model compilation on the satellite, the satellite equipment uses the embeddings of the image tiles to perform an initial analysis and identifies the approximately 1% of image tiles that are likely to contain methane leaks. The onboard methane detection model is then run on the satellite only on the image tiles that are likely to contain methane leaks. The methane detection model calculates the approximate location, size, and probability of a methane leak. The location and size of high-probability leaks are then quickly downlinked to the service system 260 and subsequently shared with oil and gas companies via the client device 271, allowing the company to more quickly deploy personnel to investigate and repair the leak.

[0054] Typically, an oil and gas company looking to detect methane plumes in satellite imagery would need to order and downlink the entire satellite imagery before determining whether all ordered images contain methane plumes. Ordering such imagery is often expensive, and it often takes hours to weeks for the imagery to be delivered to the oil and gas company for analysis. However, in this example, system 200 enables a methane detection model to run continuously onboard a spacecraft alongside other models from other clients, enabling real-time monitoring for the oil and gas company while limiting the satellite operator's resources and downlink costs.

[0055] Another example client is an energy trading hedge fund tasked with updating the fund's global dataset of oil storage tanks, which the client uses to inform trading decisions. The example analytics client does not have the funds, time, or resources to run an oil tank detection model on all available high-resolution imagery worldwide. Instead, the example client uses an example of system 200 to identify a subset of existing images that are likely to contain oil tanks, and then runs the model through the oil tank detection model. This significantly reduces the time, cost, and complexity required to update the associated dataset. This is accomplished by the example client using an example of system 200 as follows.

[0056] The client provides some reference images of oil tanks, and the serving system 260 obtains the embeddings of these reference images. The client can then initiate a catalog search of existing high-resolution imagery to identify the approximately 0.05% of images / tiles that have very similar embeddings, indicating the possible presence of oil tanks within the image. The serving system 260 then passes these identified images through a tank detection model to calculate the location and size of any oil tanks within the image. The results of the detection model are then provided to the client.

[0057] Typically, if an analyst wants to inventory all oil storage tanks worldwide, they either need to 1) run their tank detection model on the complete global satellite imagery archive, which is expensive to acquire and resource / time intensive to run, or 2) rely on external information sources to narrow their search. However, these external data sources are often incomplete and can lead to a large number of tanks being missed. In contrast, system 200 allows analysts to efficiently build a global dataset of oil tanks using a fraction of the time and resources required to run a tank detection model on all images / tiles.

[0058] Another example of a client is a coast guard that is searching for vessels engaged in illegal activities (e.g., drug trafficking, violating an oil embargo, etc.). Rapid detection of such activities will increase the chances of arrest and prosecution. An example of system 200 can be used with the coast guard as a client as follows.

[0059] Vessel detection models are deployed on a satellite constellation to provide real-time insights to Coast Guard intercept teams. Only approximately 0.1% of images / tiles captured by satellites are man-made objects in the ocean. Model orchestration, performed on the satellites, allows the approximately 0.1% of images / tiles containing man-made objects in the ocean to be identified. The satellite equipment onboard then executes the object detection model on these identified tiles to predict the class of the object present in the tile (e.g., oil platform, ship, wind turbine, etc.).

[0060] For each tile where the object detection model determines that a vessel is present, the tile is marked for further analysis. The embeddings of these tiles, along with the location of the tiles, are then quickly downlinked from the satellite to the serving system 260 for further analysis. Downlinking the embeddings and metadata is faster and more efficient than downlinking the entire image. The serving system 260 then compares the downlinked embeddings with previous images and external data. The downlinked embeddings are quickly compared with the embeddings generated for all previous images to predict the type of vessel in the image (e.g., cargo ship, tanker, cruise ship, trawler, etc.).

[0061] The detected location is also compared to an external data source (e.g., Automatic Identification System (AIS) transponder data). If the two sources disagree, for example: 1) the service system 260 appears to have identified a vessel in the image, but there is no AIS transponder record for that vessel, or 2) the tile embedding is similar to a previous embedding of a tanker, but the vessel's AIS transponder data reports that the vessel is a trawler. Discrepancies such as these indicate possible foul play and may require additional investigation by the Coast Guard.

[0062] Figure 2 A small number of systems and devices are shown for illustrative purposes. Various examples of system 200 may include Figure 2 More devices and systems shown, Figure 2 This is illustrated only as an example. Additionally, many examples of system 200 may differ from the literal examples of system 200 discussed above. For example, while the discussion of system 200 includes satellites, in various examples, system 200 may include constrained environment devices other than satellites, such as spacecraft in deep space, drones, fixed sensors such as on oil platforms, IoT devices in constrained environments, or other suitable constrained environment devices, in addition to or in lieu of satellites.

[0063] Illustrative Process

[0064] Figure 3 is a diagram illustrating an example data flow. In some examples, process 390 proceeds as follows. The steps of process 390 are performed in a service system such as Figure 2 service system 260.

[0065] First, step 391 occurs. At step 391, a plurality of reference embeddings are provided. As shown, step 392 then occurs. At step 392, a downlink session is established with the orbiting satellite. During the downlink session, steps 393 through 397 occur. As shown, step 393 then occurs. At step 393, a plurality of spacecraft embeddings are received from the orbiting satellite. Spacecraft embeddings in the plurality of spacecraft embeddings are generated on the orbiting satellite by applying an embedding generation model to corresponding image portions stored on the orbiting satellite and obtained from sensors on the orbiting satellite.

[0066] As shown, step 394 occurs next. At step 394, the spacecraft embeddings in the plurality of spacecraft embeddings are compared to the reference embeddings in the plurality of reference embeddings in vector space. As shown, step 395 occurs next. At step 395, a determination is made, based at least in part on the comparison, as to which portions of the image are high-value portions of the image. As shown, step 396 occurs next. At step 396, the portion of the image that is the high-value portion of the image is communicated to the orbiting satellite. As shown, step 397 occurs next. At step 397, the high-value portion of the image is received from the orbiting satellite. The process then proceeds to a return block where other processing is resumed.

[0067] Illustrative Equipment / Operating Environment

[0068] Figure 4 FIG. 4 is a diagram of an environment 400 in which various aspects of the present technology may be practiced. As shown, the environment 400 includes a computing device 410 and a network node 420 connected via a network 430. Figure 4Specific components of environment 400 are shown in FIG, but in other examples, environment 400 may also include additional and / or different components. For example, in some examples, environment 400 may also include a network storage device, a maintenance manager, and / or other suitable components (not shown). Figure 4 The illustrated computing device 410 can be located in various locations, including a local computer, on-premises, in the cloud, etc. For example, the computing device 410 can be on the client side, on the server side, etc.

[0069] like Figure 4 As shown, network 430 may include one or more network nodes 420 that interconnect multiple computing devices 410 and connect computing devices 410 to an external network 440, such as the Internet or an intranet. For example, network node 420 may include a switch, a router, a hub, a network controller, or other network elements. In some examples, computing devices 410 may be organized into racks, action zones, groups, sets, or other suitable partitions. For example, in the illustrated example, computing devices 410 are grouped into three host sets that are individually identified as a first host set 412a, a second host set 412b, and a third host set 412c. In the illustrated example, each host set in host sets 412a to 412c is operatively coupled to corresponding network nodes 420a to 420c, which are typically referred to as "top of rack" or "TOR" network nodes. TOR network nodes 420a to 420c can then be operatively coupled to additional network nodes 420 to form a computer network in a hierarchical, flat, mesh, or other suitable type of topology that allows communication between computing device 410 and external network 440. In other examples, multiple host sets 412a to 412c can share a single network node 420. Computing device 410 can be virtually any type of general-purpose or special-purpose computing device. For example, these computing devices can be user devices such as desktop computers, laptop computers, tablet computers, display devices, cameras, printers, or smartphones. However, in a data center environment, these computing devices can be server devices such as application server computers, virtual computing host computers, or file server computers. In addition, computing device 410 can be individually configured to provide computing, storage, and / or other suitable computing services.

[0070] In some examples, one or more of computing devices 410 are devices configured as at least part of a system for onboard sensor-related prioritization.

[0071] Illustrative Computing Devices

[0072] Figure 5is a schematic diagram illustrating one example of a computing device 500 in which various aspects of the present technology may be practiced. Computing device 500 may be virtually any type of general-purpose or special-purpose computing device. For example, computing device 500 may be a user device, such as a desktop computer, a laptop computer, a tablet computer, a display device, a camera, a printer, or a smartphone. Similarly, computing device 500 may also be a server device, such as an application server computer, a virtual computing host computer, or a file server computer. For example, computing device 500 may be Figure 4 Likewise, computer device 500 may be an example of any device illustrated or mentioned in any of the above figures, or an example of a device within any distributed system, as discussed in more detail above and below. Figure 5 As illustrated, computing device 500 may include processing circuitry 510, operating memory 520, memory controller 530, bus 540, data storage memory 550, input interface 560, output interface 570, and network adapter 580. Each of these aforementioned components of computing device 500 includes at least one hardware element.

[0073] The computing device 500 includes at least one processing circuit 510 configured to execute instructions, such as instructions for implementing the workloads, processes, and / or techniques described herein. The processing circuit 510 may include a microprocessor, a microcontroller, a graphics processor, a coprocessor, a field programmable gate array, a programmable logic device, a signal processor, and / or any other circuit suitable for processing data. The instructions mentioned above and other data (e.g., data sets, metadata, operating system instructions, etc.) may be stored in an operating memory 520 during operation of the computing device 500. The operating memory 520 may also include any of a variety of data storage devices / components, such as volatile memory, semi-volatile memory, random access memory, static memory, cache, buffer, and / or other media for storing runtime information. In one example, when the computing device 500 is powered off, the operating memory 520 does not retain information. Instead, the computing device 500 may be configured to transfer instructions from a non-volatile data storage component (e.g., data storage component 550) to the operating memory 520 as part of a boot or other loading process. In some examples, other forms of execution may be employed, such as executing directly from the data storage component 550 , eg, execute-in-place (XIP).

[0074] The operating memory 520 may include fourth-generation double data rate (DDR4) memory, third-generation double data rate (DDR3) memory, other dynamic random access memory (DRAM), high-bandwidth memory (HBM), hybrid memory cube memory, 3D stacked memory, static random access memory (SRAM), magnetoresistive random access memory (MRAM), pseudo random access memory (PSRAM), and / or other memory, and such memory may include one or more memory circuits integrated on a DIMM, SIMM, SODIMM, known good die (KGD), or other package. Such operating memory modules or devices may be organized according to channels, rows, columns, and banks. For example, the operating memory device may be coupled to the processing circuit 510 via a memory controller 530 in a channel. An example of the computing device 500 may include one or two DIMMs per channel, with one or two rows per channel. The operating memory within a row may operate using a shared clock and a shared address and command bus. Furthermore, the operating memory device may be organized into several banks, where a bank may be viewed as an array addressed by rows and columns. Based on this organization of the operating memory, physical addresses within the operating memory can be referenced by a tuple of channel, row, bank, row, and column.

[0075] Notwithstanding the above discussion, operational memory 520 does not specifically include or encompass communications media, any communications media, or any signals per se.

[0076] The memory controller 530 is configured to interface the processing circuit 510 with the operating memory 520. For example, the memory controller 530 may be configured to interface commands, addresses, and data between the operating memory 520 and the processing circuit 510. The memory controller 530 may also be configured to abstract or otherwise manage certain aspects of memory management from the processing circuit 510. Although the memory controller 530 is illustrated as a single memory controller separate from the processing circuit 510, in other examples, multiple memory controllers may be employed, the memory controller(s) may be integrated with the operating memory 520, etc. Further, the memory controller(s) may be integrated into the processing circuit 510. These and other variations are possible.

[0077] In computing device 500, data storage memory 550, input interface 560, output interface 570, and network adapter 580 are connected to processing circuitry 510 via bus 540 interface. Figure 5The bus 540 is illustrated as a single passive bus, but other configurations (such as a collection of buses, a collection of point-to-point links, input / output controllers, bridges, other interface circuitry, and / or any collection thereof) may also be used as appropriate to interface the data storage memory 550, input interface 560, output interface 570, and / or network adapter 580 to the processing circuitry 510.

[0078] In computing device 500, data storage memory 550 is used for long-term, non-volatile data storage. Data storage memory 550 may include any of a variety of non-volatile data storage devices / components, such as non-volatile memory, a disk, a disk drive, a hard drive, a solid-state drive, and / or any other medium that can be used for non-volatile storage of information. However, data storage memory 550 specifically does not include or encompass communication media, any communication media, or any signals themselves. In contrast to operating memory 520, data storage memory 550 is used by computing device 500 for non-volatile, long-term data storage, rather than for runtime data storage.

[0079] Furthermore, the computing device 500 may include or be coupled to any type of processor-readable media, such as processor-readable storage media (e.g., operating memory 520 and data storage memory 550) and communication media (e.g., communication signals and radio waves). Although the term processor-readable storage media includes operating memory 520 and data storage memory 550, the term "processor-readable storage medium" throughout the specification and claims is defined herein, whether used in the singular or plural, so that the term "processor-readable storage medium" specifically excludes and does not encompass communication media, any communication media, or any signal itself. However, the term "processor-readable storage medium" does encompass processor cache, random access memory (RAM), register memory, and the like.

[0080] The computing device 500 also includes an input interface 560, which can be configured to enable the computing device 500 to receive input from a user or other device. In addition, the computing device 500 includes an output interface 570, which can be configured to provide output from the computing device 500. In one example, the output interface 570 includes a frame buffer, a graphics processor, a graphics processor or an accelerator, and is configured to render a display to present on a separate visual display device (such as a monitor, a projector, a virtual computing client computer, etc.). In another example, the output interface 570 includes a visual display device, and is configured to render and present a display for viewing. In yet another example, the input interface 560 and / or the output interface 570 can include a universal asynchronous receiver / transmitter (UART), a serial peripheral interface (SPI), an internal integrated circuit (I2C), a general purpose input / output (GPIO), etc. In addition, the input interface 560 and / or the output interface 570 can include or be connected to any number or type of peripheral devices by an interface.

[0081] In the illustrated example, computing device 500 is configured to communicate with other computing devices or entities via network adapter 580. Network adapter 580 may include a wired network adapter, such as an Ethernet adapter, a token ring adapter, or a digital subscriber line (DSL) adapter. Network adapter 580 may also include a wireless network adapter, such as a Wi-Fi adapter, a Bluetooth adapter, a ZigBee adapter, a Long Term Evolution (LTE) adapter, SigFox, LoRa, Powerline, or a 5G adapter.

[0082] Although computing device 500 is illustrated as having certain components configured in a particular arrangement, these components and arrangements are merely one example of a computing device that may employ the present technology. In other examples, data storage memory 550, input interface 560, output interface 570, or network adapter 580 may be coupled directly to processing circuitry 510, or via an input / output controller, a bridge, or other interface circuitry. Other variations of the present technology are possible.

[0083] Some examples of computing device 500 include at least one memory (e.g., operating memory 520) having processor-executable code stored therein and at least one processor (e.g., processing unit 510) adapted to execute the processor-executable code, wherein the processor-executable code includes processor-executable instructions that, in response to execution, enable computing device 500 to perform actions, which in some examples may include actions of one or more processes described herein, such as Figure 3 The process shown is as discussed in more detail above.

[0084] The above description provides specific details to thoroughly understand and enable description of various examples of the present technology. Those skilled in the art will understand that the present technology can be practiced without many of these details. In some instances, well-known structures and functions are not shown or described in detail to avoid unnecessarily obscuring the description of the technical examples. This document intends to interpret the terms used in this disclosure in the broadest reasonable manner, even if the terms are used in conjunction with the detailed description of certain examples of the present technology. Although certain terms may be emphasized below, any terms intended to be interpreted in any restrictive manner will be clearly and specifically defined in this detailed description. Throughout the specification and claims, unless the context otherwise dictates, the following terms will at least have the meanings explicitly associated herein. The meanings identified below do not necessarily limit the terms, but only provide illustrative examples of the terms. For example, each of the terms "based on" and "based upon" is not exclusive and is equivalent to the term "based at least in part on" and includes options based on additional factors, some of which may not be described herein. As another example, the term "via" is not exclusive and is equivalent to the term "at least in part via" and includes the option of via additional factors, some of which may not be described herein. The meaning of "in" includes "in" and "on". The phrases "in one embodiment" or "in one example" as used herein do not necessarily refer to the same embodiment or example, although the phrases may. The use of a particular textual numerical identifier does not imply the presence of a numerical identifier of lesser value. For example, the specification "a small component selected from the group consisting of a third foo and a fourth bar" does not by itself mean that there are at least three foo elements, nor does it mean that there are at least four bar elements. Singular references are for clarity of reading only and include plural references unless plural references are expressly excluded. Unless specifically indicated otherwise, the term "or" is an inclusive "or" operator. For example, the phrase "A or B" means "A, B, or A and B". As used herein, the terms "component" and "system" are intended to cover various combinations of hardware, software, or hardware and software. Thus, for example, a system or component can be a process, a process executed on a computing device, a computing device, or a portion thereof. The term "cloud" or "cloud computing" refers to a shared pool of configurable computer system resources and advanced services over a wide area network (usually the Internet). "Edge" devices are devices that are not part of the cloud themselves, but serve as entry points into an enterprise or service provider's core network.

[0085] in conclusion

[0086] Although the above specific embodiments describe certain examples of the present technology and describe the expected best mode, no matter how detailed the above is in the text, the present technology can be practiced in many ways. The details can vary in the implementation, but are still covered by the technology described herein. As mentioned above, the specific terms used when describing certain features or aspects of the present technology should not be regarded as implying that the term is redefined in this article to be limited to any specific characteristics, features, or aspects associated with the term. Generally, unless such terms are clearly defined in the specific embodiments, the terms used in the following claims should not be interpreted as limiting the present technology to the specific examples disclosed herein. Therefore, the actual scope of the present technology not only covers the disclosed examples, but also covers all equivalent ways of practicing or realizing the present technology.

Claims

1. A device comprising: A device comprising at least one memory having processor-executable code stored therein and at least one processor adapted to execute the processor-executable code, wherein the processor-executable code comprises processor-executable instructions that, in response to execution, enable the device to perform actions comprising: Provide multiple reference embeddings; causing a downlink session to be established with the orbiting satellite; and During the downlink session: receiving a plurality of spacecraft embeddings from the orbiting satellite, wherein the spacecraft embeddings of the plurality of spacecraft embeddings are generated onboard the orbiting satellite by applying an embedding generation model to corresponding image portions, the corresponding image portions being stored onboard the orbiting satellite and obtained from a sensor onboard the orbiting satellite; comparing, in a vector space, the spacecraft embedding of the plurality of spacecraft embeddings with the reference embedding of the plurality of reference embeddings; making a determination based at least in part on the comparing as to which portions of the image are high-value portions of the image; communicating to the orbiting satellite which portions of the image from among the portions of the image are the high value portions of the image; and The high value portion of the image is received from the orbiting satellite.

2. The apparatus of claim 1, wherein the downlink session is a downlink session among a plurality of scheduled downlink sessions between the orbiting satellite and a ground station including the device. 3 . The apparatus of claim 1 , wherein the determining is further based on at least one of: a client request, a client search, historical client access, or a client sales pattern.

4. The apparatus of claim 1 , wherein the spacecraft embeddings in the plurality of spacecraft embeddings are feature vectors of floating point numbers.

5. A method comprising: Provide multiple reference embeddings; enabling a downlink session to be established with a constrained environment device; as well as During the downlink session: receiving a plurality of constrained environment device embeddings from the constrained environment device, wherein the constrained environment device embeddings of the plurality of constrained environment device embeddings are generated by applying an embedding generation model to corresponding image portions, the corresponding image portions being stored on the constrained environment device and obtained from a sensor on the constrained environment device; comparing, via at least one processor, the constrained environment device embedding of the plurality of constrained environment device embeddings with the reference embedding of the plurality of reference embeddings in a vector space; making a determination based at least in part on the comparing as to which portions of the image are high-value portions of the image; communicating to the constrained environment device which portions of the image from the portions of the image are the high-value portions of the image; as well as The high-value portion of the image is received from the constrained environment device.

6. A processor-readable storage medium having processor-executable code stored thereon, wherein when executed by at least one processor, the processor-executable code implements actions, including: During a downlink session with a constrained environment device: receiving a plurality of constrained environment device embeddings from the constrained environment device, wherein the constrained environment device embeddings of the plurality of constrained environment device embeddings are generated by applying an embedding generation model to corresponding image portions, the corresponding image portions being stored on the constrained environment device and obtained from a sensor on the constrained environment device; comparing, in a vector space, the constrained environment device embedding of the plurality of constrained environment device embeddings with a reference embedding of the plurality of reference embeddings; making a determination based at least in part on the comparing as to which portions of the image are high-value portions of the image; communicating to the constrained environment device which portions of the image from the portions of the image are the high-value portions of the image; as well as The high-value portion of the image is received from the constrained environment device.

7. The apparatus of claim 1, wherein the spacecraft embeddings in the plurality of spacecraft embeddings are feature vectors each having at least 256 dimensions. The apparatus of claim 1 , wherein the image comprises a plurality of satellite images.

9. The apparatus of claim 1, wherein the portion of the image is an image tile that is a uniformly sized portion of the image.

10. The apparatus of claim 1, wherein the sensor on the orbiting satellite comprises at least one of a camera, a synthetic aperture radar, a thermal imaging sensor, a hyperspectral sensor, or a video sensor.

11. The apparatus according to claim 1, wherein the embedding generation model comprises at least one of the following: an unsupervised representation learning model, a self-supervised representation learning technique, or a supervised representation learning technique.

12. The apparatus of claim 1 , wherein comparing the spacecraft embedding of the plurality of spacecraft embeddings to the reference embedding of the plurality of reference embeddings comprises: It is determined which spacecraft embeddings of the spacecraft embeddings of the plurality of spacecraft embeddings are close in the vector space to the reference embedding of the set of reference embeddings.

13. The method of claim 5, wherein the constrained environment device is at least one of the following: an Internet of Things device, an orbiting satellite, a spacecraft, or a fixed platform in a constrained environment.

14. The method of claim 5, wherein the constrained environment device is an orbiting satellite, and wherein the downlink session is a downlink session among a plurality of scheduled downlink sessions between the orbiting satellite and a ground station comprising the at least one processor.

15. The method of claim 5, wherein the determining is further based on at least one of: a client request, a client search, historical client access, or a client sales pattern. 16 . The method of claim 5 , wherein the constrained environment device embedding in the plurality of constrained environment device embeddings is a feature vector of floating point numbers.

17. The processor-readable storage medium of claim 6, wherein the constrained environment device is at least one of: an Internet of Things device, an orbiting satellite, a spacecraft, or a fixed platform in a constrained environment.

18. The processor-readable storage medium of claim 6, wherein the constrained environment device is an orbiting satellite, and wherein the downlink session is a downlink session among a plurality of scheduled downlink sessions between the orbiting satellite and a ground station comprising the at least one processor.

19. The processor-readable storage medium of claim 6, wherein the determination is further based on at least one of: a client request, a client search, historical client access, or a client sales pattern.

20. The processor-readable storage medium of claim 6, wherein the constrained environment device embedding of the plurality of constrained environment device embeddings is a feature vector of a floating point number.

Images