Distributed ocean satellite space-time asset real-time online processing method and device

Abstract

The invention provides a distributed ocean satellite space-time asset real-time online processing method. The method comprises the following steps: performing unified metadata description on ocean satellite multi-load data by adopting a general specification; unified metadata and original satellite remote sensing images are stored in an object storage bucket, and cross-regional multi-copy and version control is supported; application program interfaces are uniformly opened, RESTful endpoints conforming to specifications are provided, and attribute, time and space composite retrieval of stored metadata is achieved; calling an API (Application Program Interface) to obtain data in the storage bucket, and cutting to generate tiles according to requirements by utilizing an image pyramid and a GDAL (Graphics Data Allocation Language) virtual file system on the basis of metadata description; and automatically expanding the capacity based on the tile generation condition, and storing the hotspot tiles through a distributed cache. The invention further provides a device capable of executing the method and a computer readable storage medium. Compared with a cloud data management scheme, the method provided by the invention has higher standardization and QPS peak expansion capability and lower redundant storage and tile request delay time.

Description

Technical Field

[0001] The present invention relates to the technical fields of satellite remote sensing data management, cloud-native microservices, and geographic information system (GIS), and particularly relates to a distributed spatio-temporal asset real-time online processing method, device, and computer-readable storage medium for multi-payload and multi-resolution remote sensing data of ocean satellites. Background Art

[0002] Satellite spatio-temporal assets are data and information obtained by satellites with spatio-temporal attributes, including various data types, such as satellite remote sensing images, hyperspectral optical data, synthetic aperture radar (SAR) data, etc. Satellite spatio-temporal assets have wide application values in multiple fields. Through the analysis and processing of satellite spatio-temporal assets, decision-making support can be provided for resource investigation and monitoring, environmental monitoring and protection, urban planning and management, agricultural yield estimation and disaster assessment, etc.

[0003] The spatio-temporal assets of ocean satellites contain a large number of remote sensing images, which have characteristics such as high temporal and spatial resolutions, long time series, large spatial coverage, and multi-dimensional spatial information. With the successful launch and in-orbit operation of China's new generation of ocean color observation satellites such as HY-1D and HY-1E, a large amount of remote sensing images are transmitted back to the ground at a minute-level rate. For the data processing of the massive satellite remote sensing images, the existing cloud data management solutions are mainly based on offline pre-slicing, static publishing, or private interfaces, and have the following deficiencies: (1) Lack of a unified metadata standard: The metadata formats between different satellites and different payloads are inconsistent, affecting data sharing and secondary development; (2) Insufficient real-time performance: Offline slicing needs to be pre-generated, resulting in 20-40% redundant storage and being difficult to meet near-real-time (NRT) business scenarios; (3) Poor scalability: Traditional monolithic or fixed-node deployment modes are difficult to elastically scale in high-concurrency scenarios; (4) Closed interfaces: Lack of open APIs based on international standards, restricting the access efficiency of ecological partners and scientific research users.

[0004] Therefore, it is necessary to propose a real-time online processing method applicable to distributed ocean satellite data to solve the above industrial application problems. Summary of the Invention

[0005] Aiming at the above problems existing in the prior art, the primary object of the present invention is to provide a distributed ocean satellite data management and online processing method that follows the Spatio-Temporal Asset Catalog (STAC) standard, and through a cloud-native architecture and dynamic slicing technology, realizes the unified description, elastic storage, real-time retrieval, and millisecond-level visual access of massive ocean satellite remote sensing images.

[0006] Another object of the present invention is to provide a device, an electronic device, and a computer-readable storage medium capable of executing the above method.

[0007] The above object of the present invention is achieved by the following technical solutions: In a first aspect, the present invention provides a distributed real-time online processing method for spatio-temporal assets of ocean satellites, including: Step S1. Metadata standardization Use the current general specifications to perform metadata description of the unified data exchange format for multi-payload data of ocean satellites.

[0008] Step S2: Distributed cloud object storage Simultaneously store the metadata in the unified data exchange format obtained in step S1 and the original satellite remote sensing images in an object storage bucket (Bucket), supporting cross-region multi-copy and version control.

[0009] Step S3: Unified open application programming interface (API) Provide RESTful endpoints that conform to the general specifications of geospatial data to implement the attribute, time, and space composite retrieval of the metadata stored in step S2.

[0010] Step S4: Dynamic web map tile service (WMTS) Call the API provided in step S3 to obtain the data in the storage bucket in step S2. Based on the containerized microservice architecture, based on the metadata description in step S1, use the image pyramid and GDAL virtual file system (VSI) to generate 256×256 PNG / JPEG tiles on demand.

[0011] Step S5: Elastic expansion and high-frequency caching Automatically expand the capacity based on the situation of generating tiles in step S4, and store the hot tiles in step S4 through distributed caching (Redis / Memcached).

[0012] In the method of the present invention, the current general specifications described in step S1 include STAC 1.0.0 and EO, Projection, and Sat extensions.

[0013] In the method of the present invention, the unified data exchange format described in step S1 is specifically JSON format.

[0014] In the method of the present invention, the metadata described in step S1 includes basic attributes, time attributes, spatial ranges, projection information, payload information, and asset information.

[0015] In the method of the present invention, the original satellite remote sensing image described in step S2 is the original HDF5 image.

[0016] In the method of the present invention, the support for cross-region multi-copy and version control described in step S2 specifically means that object storage supports cross-availability zone three-copy redundancy, lifecycle management, and version control.

[0017] In the method of the present invention, the general specification for geospatial data described in step S3 includes OGC STA (Open Geospatial Consortium Spatio-Temporal API) and STAC.

[0018] In the method of the present invention, the STAC API described in step S3 at least includes / stac, / collections, / items, / stac / search endpoints.

[0019] In the method of the present invention, the dynamic web map tile service described in step S4 uses image pyramid and GDALVRT technologies to generate tiles.

[0020] In the method of the present invention, the automatic scaling described in step S5 specifically means that based on Kubernetes HPA, the number of instances is automatically adjusted according to CPU usage or request rate.

[0021] In the method of the present invention, storing hot tiles through distributed caching (Redis / Memcached) described in step S5 specifically means using Redis distributed caching to store the tile data with the highest access frequency in the last 24 hours.

[0022] Further, in order to improve data security and compliance, the method of the present invention further includes any one or a combination of two or more of the following: encrypting the object storage process described in step S2, performing token authentication on the API described in step S3, and / or encrypting tile transmission for the tile service described in step S4.

[0023] In a second aspect, the present invention also provides a distributed online processing device capable of executing the method of the first aspect of the present invention, including: a metadata service module, an object storage module, a tile service module, an authentication module, and a monitoring module.

[0024] The metadata service module is deployed in a PostgreSQL + PostGIS cluster; it is used for unified format metadata description of ocean satellite multi-payload data.

[0025] The object storage module is used to store the original HDF5 image in an object storage bucket (Bucket); The slicing service module uses the FastAPI framework and is encapsulated as a container image to run in a Kubernetes cluster, and is used to generate 256×256 PNG / JPEG tiles on demand by using an image pyramid and the GDAL virtual file system (VSI).

[0026] The authentication module implements Token-level access control based on the OAuth2 protocol; The monitoring module is used to monitor distributed caches, cluster server resources, and storage resources, and prompt possible overload situations, so as to dynamically increase resources before resource exhaustion.

[0027] Thirdly, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect of the present invention are implemented.

[0028] The present invention discloses a real-time processing method, device, and computer-readable storage medium for ocean satellite spatio-temporal assets based on STAC metadata and cloud-native architecture. The method of the present invention realizes millisecond-level slice response for massive remote sensing images through HDF5 pyramid - tiled index (TPI), multi-level caching, and Prometheus-driven automatic scaling, and can be widely applied to near-real-time scenarios such as ocean water color monitoring, red tide early warning, fishing ground environment assessment, nearshore pollution source tracing, and various business scenarios such as global carbon flux inversion.

[0029] Compared with the existing cloud data management solutions, the method of the present invention has the following beneficial effects: 1. Standardization: STAC metadata and open APIs improve cross-platform interoperability and reduce integration costs by 30%; 2. Reduced storage: Dynamic slicing cancels offline pre-slicing, significantly reducing redundant storage by more than 60%; 3. High concurrency: The elastic expansion of Kubernetes increases the QPS peak expansion ability to more than 3 times that of the original solution; 4. Ultra-fast response: The pyramid + cache strategy controls the latency of 95% of tile requests within 150ms; 5. Secure and reliable: It meets the requirements of the third level of "Information Security Level Protection 2.0" and the "Measures for the Security Assessment of Cross-border Data Transfers". BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall system architecture described in the specific embodiment of the present invention (composed of a metadata service, a storage service, a slicing service, an authentication service, and a monitoring service).

[0031] Figure 2It embodies the dynamic WMTS slice generation process in the specific embodiment of the present invention.

[0032] Figure 3 It is an example structure of STAC metadata.

[0033] Figure 4 It is the load balancing and automatic scaling logic diagram in the system described in the specific embodiment of the present invention. Specific Embodiment

[0034] The technical solution of the present application will be further described in detail below through the description of specific embodiments. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements, modules or steps, or elements, modules or steps with the same or similar functions from beginning to end. The specific embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0035] The present invention provides a distributed real-time online processing system for spatio-temporal assets of ocean satellites, and its overall structure is as Figure 1 shown, including a browsing layer, a gateway layer and a service layer.

[0036] The browsing layer is provided with a user browser, which is used to obtain data in real time at the user front end and display the data in the form of map slices; The gateway layer is provided with an API gateway, which is used to provide a unified access URL externally, and transfer the access request to different cluster processing units (PODS of K8S) through the gateway; The service layer is provided with a slice service module, a metadata service module and an object storage module; the slice service provides real-time data slicing service, directly returns the data to the front end in the form of memory bytes, avoiding the traditional slices occupying a large amount of storage space, the metadata service quickly locates the data file and access location according to the slice request of the user front end, and the object storage module provides data transmission interaction based on the HTTPS protocol, avoiding the conversion of data on different transmission protocols.

[0037] The present invention also provides a distributed real-time online processing method for spatio-temporal assets of ocean satellites, which can be specifically implemented in the following manner: S1. Generate metadata containing 64-bit Z-Key for each satellite remote sensing image based on STAC 1.0.0; S2. Store the satellite remote sensing image and the metadata produced in S1 in the order of Z-Key in HDF5 Chunk = 256px × 256px × Bands, and generate an index tpi.idx; S3. Calculate the Z-Key when the WMTS request arrives, use tpi.idx to directly reach the offset, call pread() to read the Chunk and encode it with PNG Deflate; generate a tile; S4. Write the tile generated in S3 into the Redis distributed cluster (Redis Cluster) and return; S5. Use Prometheus and Kubernetes HPA to automatically scale out the instances of the tile server when the metric rate(tile_server_request_duration_seconds_count[2m]) > 200 QPS.

[0038] In a preferred embodiment of the present invention, S3 is implemented by the following method: HDF5 pyramid - tiled index (TPI), using the TIP algorithm, interleaves the tile row and column through Z - Curve encoding into a 64 - bit key and stores it in the tpi.idx mapping <offset, size>. During runtime, directly perform pread for accurate skip reading of the target Chunk, and the single - slice IO latency is less than 5 ms. The specific implementation requirements for the payload COCTS are shown in Table 1 below.

[0039] Table 1 Resolution (m) Map size (px) Chunk size Offset time consumption 1100 8640×4320 2048×2048×Bands 11µs 2200 4320×2160 1024×1024×Bands 9µs 4400 2160×1080 512×512×Bands 6µs 8800 1080×540 256×256×Bands 4µs In a preferred embodiment of the present invention, S4 adopts a multi - level caching strategy: Redis Cluster + object storage bucket cache, with a required hit rate of 93% / 20%.

[0040] In a preferred embodiment of the present invention, S5 adopts the HPA scaling strategy as Figure 4 shown: Grab the metric tile_server_request_duration_seconds_count through Prometheus and calculate the two - minute sliding window rate in PromQL; when the custom metric QPS > 200, HPA triggers the scaling - out strategy (increase the number of Pod replicas), and when QPS < 80 and the current number of replicas exceeds the minimum threshold, HPA performs scaling - in to reclaim computing resources. The performance achieved by this algorithm is shown in Table 2 below: Table 2 Index The present invention Traditional solution Improvement Peak QPS 8200 2630 ↑212% Average latency (ms) 95 210 ↓54% Storage occupancy (TB, 2y) 78 212 ↓63% Furthermore, Figure 2 the following exclusive algorithm is adopted for the process shown 1. Tile-Pyramid Index (TPI) Construction Process The TPI aims to locate any WMTS tile request to the internal data block of the HDF5 file within O(1) time. Its core idea is: first construct multi-level pyramids by resolution, then interleaved map the two-dimensional tile numbers to 64-bit integer Z-Key, and finally write the <offset, size> key-value pairs of Z-Key to the index file tpi.idx.

[0041] Step1: Execute gdaladdo on the original image to generate pyramid files with resolutions L0 - L3 (using MAX without resampling). Step2: Repack the HDF5 Chunks of each layer with a fixed block size of 256×256×Bands; Step3: Traverse all Chunks of each level and calculate: ZKey = bit_interleave(TileRow, TileCol) / / Z-curve Step4: Record the entry <ZKey, offset64, size32, crc32> to the memory buffer; Step5: Write to the binary index file tpi.idx in ascending order of ZKey, and append 8 bytes of Magic (0xA1B2C3D4E5F60708) at the end. 2. Chunk Location and Cropping Algorithm during Execution Input: WMTS request (z, x, y), index file tpi.idx, HDF5 image file.

[0042] R1. Calculate TileRow = y, TileCol = x at level z; R2. Call z_encode(TileRow, TileCol) to obtain ZKey; R3. Use memory-mapped tpi.idx (mmap) + binary search with ≤ 18 comparisons to locate the entry; R4. Use pread(fd, buf, size, offset) to read the Chunk into shared memory; R5. Perform 256×256 ROI cropping with GPU or SIMD acceleration; R6. Use libpng 1.6 Deflate-fast compression, with an average time consumption of 3.1ms; R7. Write to Redis Cluster and return to the client.

[0043] 3. Hybrid-LFU / LRU cache eviction algorithm High-resolution Tile access shows a characteristic of concentrated hotspots, and LFU is adopted; low-resolution multi-roaming access shows a characteristic of temporal locality, and LRU is adopted. Weight function: w = 0.6*hit_count_norm + 0.4*recency_norm. When the cache_size is approximately the set value, the object with the smallest weight is obtained and deleted from redis.

[0044] 4. Prometheus-HPA metric bridging algorithm B1. Prometheus scrapes tile_server_request_duration_seconds_count every 15s; B2. Prometheus-adapter executes PromQL: rate(...[2m]); B3. Write the result to the Kubernetes CustomMetrics API (metric: ts_qps); B4. HPA metric rule: When ts_qps > 200, replicas += 2 automatically; when ts_qps < 80 and replicas > 2, it is halved; B5. After the scale-out is completed, Prometheus re-scrapes the metrics of the new Pods to form a closed loop.

[0045] STAC example of HY1D satellite COCTS (its structure is as Figure 3 shown) is as follows: { "type": "Feature", "hy_version": "1.0.0", "id": "HY-1D_COCTS_20250318T023456Z_0123", "collection": "hy-1d_cocts", "bbox": [120.0, 30.0, 131.0, 42.0], "geometry": { "type": "Polygon", "coordinates": [120.0, 30.0], [131.0, 30.0], [131.0, 42.0], [120.0, 42.0], [120.0, 30.0] }, "properties": { "datetime": "2025-03-18T02:34:56Z", "platform": "HY-1D", "instruments": ["COCTS"], "constellation": "HY", "mission": "HY-1D", "gsd": 1100, / / Ground sampling distance is about 1.1 km "cloud_cover": null, / / Cloud cover is usually not calculated for ocean color products and can be set to null "absolute_orbit": 1456, "relative_orbit": 37, "orbit_state": "ascending" }, "proj:epsg": 4326, "assets": { "B01": { "href": "https: / / nsoas.org / hy-1d / B01.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Coastal aerosol (B01)", "eo:bands": ​​​{ "name": "B01", "common_name": "coastal", "center_wavelength": 0.412, "full_width_half_max": 0.020} }, "B02": { "href": "https: / / nsoas.org / hy-1d / B02.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Blue-1 (B02)", "eo:bands": { "name": "B02", "common_name": "blue", "center_wavelength":0.443, "full_width_half_max": 0.020} }, "B03": { "href": "https: / / nsoas.org / hy-1d / B03.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Blue-2 (B03)", "eo:bands": { "name": "B03", "common_name": "blue", "center_wavelength":0.490, "full_width_half_max": 0.020} }, "B04": { "href": "https: / / nsoas.org / hy-1d / B04.hdf", "type": "application / x-hdf", "roles": ["data"], ​​​"title": "Green (B04)", "eo:bands": { "name": "B04", "common_name": "green", "center_wavelength":0.555, "full_width_half_max": 0.020} }, "B05": { "href": "https: / / nsoas.org / hy-1d / B05.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Red (B05)", "eo:bands": { "name": "B05", "common_name": "red", "center_wavelength":0.670, "full_width_half_max": 0.020} }, "B06": { "href": "https: / / nsoas.org / hy-1d / B06.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Narrow NIR (B06)", "eo:bands": { "name": "B06", "common_name": "nir08", "center_wavelength":0.753, "full_width_half_max": 0.010} }, "B07": { "href": "https: / / nsoas.org / hy-1d / B07.hdf", ​​​"type": "application / x-hdf", "roles": ["data"], "title": "Broad NIR (B07)", "eo:bands": { "name": "B07", "common_name": "nir09", "center_wavelength":0.865, "full_width_half_max": 0.040} }, "B08": { "href": "https: / / nsoas.org / hy-1d / B08.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Shortwave Infrared (SWIR 1) (B08)", "eo:bands": { "name": "B08", "common_name": "swir12", "center_wavelength": 1.250, "full_width_half_max": 0.020} }, "B09": { "href": "https: / / nsoas.org / hy-1d / B09.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Shortwave Infrared (SWIR 2) (B09)", "eo:bands": { "name": "B09", "common_name": "swir16", "center_wavelength": 1.640, "full_width_half_max": 0.060} ​​​}, "B10": { "href": "https: / / nsoas.org / hy-1d / B10.hdf", "type": "application / x-hdf", "roles": ["data"], "title": "Shortwave Infrared (SWIR 3) (B10)", "eo:bands": { "name": "B10", "common_name": "swir22", "center_wavelength": 2.170, "full_width_half_max": 0.080} } } } Embodiments of the present application also provide an electronic device, which includes a device for implementing the foregoing method. The electronic device exists in various forms, including but not limited to: (1) Ultra-mobile personal computer device: This type of device belongs to the category of personal computers, has computing and processing functions, and generally also has networking characteristics. This type of terminal is generally a computer for processing computing data.

[0046] (2) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc.

[0047] (3) Other electronic devices with data computing and processing capabilities.

[0048] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0049] ​​In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0050] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A real-time online processing method for distributed ocean satellite spatio-temporal assets, characterized in that, Including: Step S1. Metadata standardization Use current general specifications to perform metadata description in a unified data exchange format for multi-payload data of ocean satellites; The metadata includes basic attributes, time attributes, spatial range, projection information, payload information, and asset information; Step S2: Distributed cloud object storage Store the metadata in the unified data exchange format obtained in Step S1 and the original satellite remote sensing images in the object storage bucket, supporting cross-region multi-copy and version control; Step S3: Unified open application programming interface Provide RESTful endpoints that conform to the general specifications of geospatial data to achieve attribute, time, and space compound retrieval of the metadata stored in Step S2; Step S4: Dynamic web map slicing service Call the API provided in Step S3 to obtain the data in the storage bucket in Step S2. Based on the containerized microservice architecture and the metadata description in Step S1, use image pyramids and the GDAL virtual file system to generate 256×256 PNG / JPEG tiles on demand; Step S5: Elastic expansion and high-frequency caching Automatically expand the capacity based on the situation of generating tiles in Step S4, and store the hot tiles in Step S4 through distributed caching; the specific automatic expansion is based on Kubernetes HPA, and automatically adjusts the number of instances according to the CPU usage rate or request rate; the specific method of storing hot tiles through distributed caching is to use Redis distributed caching to store the tile data with the highest access frequency in the last 24 hours.

2. The method according to claim 1, characterized in that: The current general specifications described in Step S1 include STAC 1.0.0 and EO, Projection, Sat extensions.

3. The method according to claim 1, characterized in that: The unified data exchange format described in Step S1 is specifically the JSON format.

4. The method according to claim 1, characterized in that: The original satellite remote sensing image described in Step S2 is the original HDF5 image.

5. The method according to claim 1, characterized in that: The support for cross-region multi-copy and version control described in Step S2 is specifically that object storage supports three-replica redundancy across availability zones, lifecycle management, and version control.

6. The method according to claim 1, wherein: The general specifications of geospatial data described in Step S3 include OGC STA and STAC.

7. The method according to claim 1, characterized in that: The STAC API described in Step S3 includes at least / stac, / collections, / items, / stac / search endpoints.

8. The method according to any one of claims 1-7, characterized in that: The method further includes any one or a combination of two or more of the following: encrypting the object storage process described in Step S2, performing token authentication on the API described in Step S3, and / or encrypting tile transmission for the slicing service described in Step S4.

9. A distributed online processing device capable of executing the method according to claim 8, characterized in that, Including: Metadata service module, deployed in a PostgreSQL + PostGIS cluster; used to perform metadata description in a unified format for multi-payload data of ocean satellites; Object storage module, used to store the original HDF5 images in the object storage bucket; Slicing service module, using the FastAPI framework and encapsulated as a container image to run in a Kubernetes cluster, used to generate 256×256 PNG / JPEG tiles on demand using image pyramids and the GDAL virtual file system; The authentication module implements Token-level access control based on the OAuth2 protocol; The monitoring module is used to monitor distributed caches, cluster server resources, and storage resources, and prompt possible overload situations.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the steps of the method according to any one of claims 1-8 are implemented.

Images