Remote sensing image data processing method and system based on Spark architecture

Abstract

The invention relates to a remote sensing image data processing method and system based on a Spark architecture, and belongs to the technical field of remote sensing image data processing. The method comprises the following steps: converting a collected original file into a COG format, constructing a hybrid index through a spatial dimension reduction technology, extracting metadata and establishing a metadatabase; based on the hybrid index, quickly positioning a target tile through spatial query, reading a data block, and constructing a TileLayerRDD of a spatial partition in a Spark cluster; and dynamically assembling and calculating the directed acyclic graph according to the algorithm warehouse template, and submitting the directed acyclic graph to a Spark cluster for distributed memory calculation. According to the method, rapid retrieval is achieved through the COG format and the mixed index, pre-slicing is replaced with Spark distributed computing, the storage pressure is greatly reduced, and on-demand real-time analysis and high-concurrency online service of mass remote sensing data are finally achieved through micro-service and elastic resource scheduling.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image data processing technology, specifically relating to a remote sensing image data processing method and system based on the Spark architecture. Background Technology

[0002] With the development of Earth observation technology, the amount of remote sensing image data has increased dramatically. Traditional remote sensing image data processing methods rely heavily on single-machine storage and computing, and face significant performance bottlenecks.

[0003] For the ever-increasing volume of high-resolution, long-term remote sensing data, traditional solutions typically employ pre-tiling to generate static pyramids, a process that is not only time-consuming and labor-intensive, but also consumes enormous amounts of storage space, and the data is difficult to update once published. In the analysis phase, users need to download the entire imagery to their local machine and use specialized desktop software for band calculations and information extraction, a cumbersome process that cannot meet the urgent needs for real-time computing and on-demand services in scenarios such as emergency monitoring and interactive analysis. Existing web-based GIS systems are also mostly limited to map visualization; their computing engines struggle to support parallel computing and high-concurrency access to large-scale raster data, resulting in a severe deficiency in the intelligent extraction and agile service capabilities of remote sensing data. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides a remote sensing image data processing method and system based on the Spark architecture. It achieves rapid retrieval through COG format and hybrid indexing, utilizes Spark distributed computing to replace pre-slicing, significantly reducing storage pressure, and ultimately realizes on-demand real-time analysis and high-concurrency online services for massive remote sensing data through microservices and elastic resource scheduling.

[0005] The objective of this invention can be achieved through the following technical solutions: The first aspect of this disclosure provides a method for processing remote sensing image data based on the Spark architecture, comprising the following steps: S1. Data preprocessing and index building: Convert the collected raw files into COG format, build a hybrid index through spatial dimensionality reduction technology, extract metadata and build a metadata database; S2, Distributed Data Organization: Based on hybrid indexes, target tiles are quickly located through spatial queries, data blocks are read and spatially partitioned TileLayerRDDs are built in the Spark cluster, while a persistent caching mechanism is introduced. S3, Parallel computation of image data: Dynamically assemble and compute directed acyclic graphs based on algorithm repository templates, submit them to Spark clusters for distributed in-memory computation, and achieve real-time elastic scaling of computing resources through Kubernetes; S4, Remote Sensing Big Data Platform Service Release: Through a microservice architecture, distributed computing is encapsulated as a RESTful API, supporting users to submit processing requests in real time. After dynamically completing calculations, projections, and caching, thematic maps and statistical results are returned instantly.

[0006] Furthermore, the data preprocessing and index construction includes the following steps: S11. Standardized data import: Use the GDAL command-line tool or write automated scripts to batch convert the collected raw format data into COG files; S12. Spatial Dimensionality Reduction Index: Morton codes are generated by encoding COG tiles using Z-curves, a hybrid index is constructed, and metadata is extracted and stored using GDAL or Geotrellis.

[0007] Furthermore, the spatial dimensionality reduction index includes the following steps: Z-curve encoding: For each tile in the COG file, a unique, one-dimensional Morton code is calculated using the Z-curve algorithm based on the row and column numbers within the entire image range. Build a hybrid index: Construct a structured string with an index key in the format {spatial code}-{timestamp}-{attribute label}; Metadata extraction and storage: Use the GDAL or Geotrellis library to parse COG files and extract key metadata, including spatial extent, projection information, number of bands, acquisition time, and cloud cover. The extracted metadata is associated with the generated hybrid index key and stored in a separate column family of metadata.

[0008] Furthermore, the distributed data organization includes the following steps: S21. Distributed storage and data association: Upload the pre-processed COG files to HDFS for persistent storage, and create a communication entry for each COG file, including the physical address and hybrid index key; S22. Construct TileLayerRDD: Locate AOI tiles using a hybrid index, read data after parsing the file path and tile offset, and construct TileLayerRDD in the Spark cluster; S23. Introduce a smart caching mechanism: For TileLayerRDDs that are frequently accessed and computed, call the rdd.persist method to cache them in the memory or SSD of the Spark cluster.

[0009] Furthermore, the construction of TileLayerRDD includes the following steps: For a specific AOI region, first query the metadata database and use the hybrid index to quickly locate the list of hybrid index keys corresponding to all tiles covering the current AOI region; For each key in the mixed index key list, Geotrellis queries the metadata to parse the COG file path and tile offset corresponding to each key, and then reads the tile data block through HadoopValueReader; The read tile data blocks are sent to the Spark cluster, and TileLayerRDD objects are constructed based on the spatial key information inherent in the tiles. Spark's partitioner allocates spatially adjacent tiles to the same compute node.

[0010] Furthermore, the image data is processed in parallel, including the following steps: S31. Dynamic assembly of computational tasks: Configure the algorithm repository, encapsulate NDVI, NDBI and MNDWI remote sensing indices into standard Geotrellis operators, and dynamically construct and execute a directed acyclic graph. S32, Distributed Execution: Based on the execution directed acyclic graph, the Spark scheduler assigns each partition of the bound TileLayerRDD to a compute node. On each node, Geotrellis operators apply user-defined operations as closure functions to each tile of data within the partition. S33. Resource Elastic Scheduling: Deploy the Spark computing cluster on the Kubernetes container orchestration platform, configure an elastic controller to continuously monitor the task queue length or resource load of the Spark driver behind the microservice gateway, and execute scheduling policies.

[0011] Furthermore, the dynamic construction of the directed acyclic graph includes: Request parsing: When the microservice layer receives a request, it first retrieves the corresponding calculation template by name from the algorithm repository; Logical plan generation: The backend service parses out the bands that need to be extracted from TileLayerRDD based on the calculation template and generates a logical execution plan; Physical plan binding: Binds the logical plan to a currently ready TileLayerRDD that contains concrete data; In the aforementioned elastic resource scheduling, the scheduling strategies include: When a backlog of tasks or a shortage of computing resources is detected, the controller automatically and quickly increases the number of Spark Executor Pod instances via the Kubernetes API. When the system is idle, the controller will automatically reduce the number of redundant Executors and release the idle computing resources to other services in the cluster.

[0012] Furthermore, the release of the remote sensing big data platform service includes the following steps: S41. Create a microservice architecture and API: Divide the complex remote sensing processing system into a set of independent microservices according to functional modules; deploy an API gateway in front of all microservices and configure routing rules to map different request paths to the corresponding backend microservices; at the same time, enable and configure authentication, access control and request rate limiting policies on the gateway. S42. Service Chain Collaboration and Backpressure Mechanism: When the backend processing service receives a computation request forwarded by the API gateway, it activates the service chain for collaboration, including: Parsing and Querying: The processing service first parses the request parameters, then initiates a query to the metadata database, and uses the constructed hybrid index to quickly locate the list of data tiles that need to be processed; Task assembly and submission: The service dynamically assembles the computation process from the algorithm repository according to the requested algorithm, and then submits a distributed computing job to the Spark cluster; Implement a backpressure mechanism for system self-protection, continuously monitoring the resource utilization and pending task queue length of the Spark cluster; when monitoring metrics show that cluster resources have reached the saturation threshold, the backpressure mechanism is automatically triggered. Reduce request reception rate: Limit the reception rate of new computing requests through the API gateway; Queuing or rejecting requests: Requests exceeding processing capacity are buffered by being directed to a message queue, or a "system busy" message is returned directly to prevent a flood of requests from overwhelming the computing cluster; S43. Dynamic Output and Caching of Multiple Coordinate Systems: Reads the target coordinate system, calls the distributed reprojection operator to transform the TileLayerRDD, and generates results in the specified coordinate system; simultaneously, a caching mechanism is built to query / store calculation results via composite keys, including the following steps: Perform dynamic coordinate projection: Identify output coordinate system parameters: Read and parse the target coordinate system parameters specified in the user request; Triggering distributed reprojection computation: For requests that require coordinate system transformation, the Geotrellis reproject operator is called to initiate a distributed, parallel coordinate transformation operation on the TileLayerRDD that carries the computation results; Generate multi-coordinate system results: Produce new TileLayerRDD results that conform to the user-specified coordinate system, enabling instant conversion of the same data source to multiple standard map services; Build and enable a projection result caching mechanism: Design cache key generation rules: Develop a composite key generation rule based on algorithm name, hash value of requested region spatial range, and target coordinate system to generate a globally unique cache identifier for each computation request; Implement cache query and write: Before returning the final result, first query the cache database using the generated cache key; if it exists, return the cached result directly; if it does not exist, execute the complete calculation and projection process, and after the process is completed, associate the final result with the cache key and store it in the cache database; Configure cache management strategies: Configure cache lifetime and eviction policies to ensure the freshness of cached data and the effective use of storage space; S44. Result Return: Render the final TileLayerRDD into a PNG format thematic map image; simultaneously perform statistical operations quickly on the distributed cluster; The backend service packages the generated thematic map images and statistical results, and returns them to the frontend browser via API. After receiving the results, the frontend application clearly displays the thematic map to the user.

[0013] The second aspect of this disclosure provides a remote sensing image data processing system based on Spark architecture, which executes a remote sensing image data processing method based on Spark architecture as described above, including a data preprocessing and index building module, a distributed storage and data organization module, a parallel computing and task scheduling module, a service publishing and interface management module, a front-end interaction and visualization module, and a system monitoring and resource management module. The data preprocessing and index building module is responsible for preprocessing the original remote sensing image data, building a multi-dimensional hybrid index, and extracting and storing the image metadata into the metadata database to provide a data foundation for subsequent rapid retrieval and positioning. The distributed storage and data organization module is used to store the preprocessed COG files into the HDFS distributed file, establish a mapping relationship between index keys and physical file paths through the metadata directory, quickly locate and construct TileLayerRDD according to the AOI area requested by the user, and introduce an intelligent caching mechanism to persist frequently accessed RDDs in memory or SSD. The parallel computing and task scheduling module is used to provide an algorithm repository with pre-set standard operators, build and submit a directed acyclic graph to the Spark cluster to realize distributed in-memory computing, use Kubernetes to realize elastic scaling of computing resources, and dynamically adjust the number of Executors according to the task load.

[0014] As a preferred technical solution of the present invention, the service publishing and interface management module adopts a microservice architecture. It handles authentication, authorization and routing uniformly through the API gateway. The backend service chain is responsible for parsing requests, locating data and dynamically submitting calculation jobs. It automatically limits or queues requests by monitoring the cluster load. At the same time, the calculation results support dynamic coordinate reprojection and generate a cache key together with the request parameters. For duplicate requests, the cached results are directly returned. The front-end interaction and visualization module is used to realize the interaction of geographic information, the initiation of analysis tasks, and the dynamic visualization of calculation results by building a web user interface; The system monitoring and resource management module is used to continuously collect various indicators from infrastructure to application services by deploying a comprehensive monitoring agent, perform unified management of user permissions, access logs and resource quotas, and trigger alarms when anomalies or resource bottlenecks are detected.

[0015] The beneficial effects of this invention are as follows: This invention firstly achieves rapid retrieval and region of interest localization of massive image data through the combined use of COG and spatial-temporal-attribute hybrid indexes, greatly improving data access efficiency. Secondly, based on a distributed in-memory computing core built with Spark and Geotrellis, and combined with elastic resource scheduling implemented by Kubernetes, complex remote sensing index calculations can be responded to quickly, realizing the transformation from "data download and processing" to "real-time cloud computing". Finally, through microservice encapsulation and dynamic computing chain assembly, a standardized and scalable RESTful API is provided to support high-concurrency access by multiple users, and the stability and high performance of the service are ensured through backpressure and caching mechanisms. Ultimately, it provides users with an integrated, real-time online service experience from data browsing and on-demand analysis to result visualization. Attached Figure Description

[0016] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0017] Figure 1 A schematic diagram illustrating the steps of a remote sensing image data processing method based on the Spark architecture provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the data preprocessing and index construction steps provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the steps of distributed data organization provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the steps of parallel computation of image data provided in an embodiment of the present invention; Figure 5A schematic diagram illustrating the steps of publishing remote sensing big data platform services provided in this embodiment of the invention; Figure 6 This is a schematic diagram of the structure of a remote sensing image data processing system based on the Spark architecture, provided as an embodiment of the present invention. Detailed Implementation

[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0019] This embodiment provides a remote sensing image data processing method based on the Spark architecture, such as... Figure 1 As shown, it includes the following steps: S1. Data Preprocessing and Index Building: The collected raw files are converted to COG format, a hybrid index is built using spatial dimensionality reduction techniques, metadata is extracted, and a metadata database is established, such as... Figure 2 As shown, it includes the following steps: S11. Standardized Data Import: Use the GDAL command-line tool or write automated scripts to batch convert the collected raw GeoTIFF, IMG and other format data into COG (Cloud Optimized GeoTIFF) files.

[0020] It should be noted that GeoTIFF is an OGC standard built upon the TIFF ISO standard, used for storing data as a two-dimensional raster layer. To address the read / write performance issues of GeoTIFF files in object storage, the original file was modified to convert it into a COG file, enabling it to be published on an HTTP server and efficiently extract sub-region information.

[0021] COG has two main characteristics. First, it uses tiles to organize data, changing the data organization from strip-based to tile-based. Images are divided into continuous, regular blocks, typically 512x512 pixels in size, and stored regularly in files. Second, it generates a series of overview layers by resampling the original data to accommodate data acquisition needs at different scales. COG imagery has a pyramid structure and can be partially retrieved using HTTP GET range requests, making it well-suited for data transmission in network environments. The GDAL virtual format mechanism can retrieve image metadata, thumbnails, and specified spatial subsets of data (single / multispectral bands). When accessing data tiles of different resolution sizes, this mechanism selects the appropriate overview level for data reading, thereby reducing the server's data reading load and improving system response speed. Optimized overview metadata information allows for rapid location and reading of local image data blocks within a specified range, significantly improving the reading performance of massive amounts of remote sensing image data.

[0022] Understandably, the GDAL command-line tool is a collection of utilities installed with the GDAL library and executed in a terminal (such as Linux's bash, Windows' Command Prompt, or PowerShell). It allows users to perform various raster and vector geospatial data transformation, processing, and analysis tasks directly by entering commands without writing any code.

[0023] S12. Spatial Dimensionality Reduction Index: Morton codes are generated by encoding COG tiles using Z-curves, a hybrid index is constructed, and metadata is extracted and stored in the database using GDAL or Geotrellis, including the following steps: Z-curve encoding: For each tile (typically 512×512 pixels) in the COG file, a unique, one-dimensional Morton code is calculated using the Z-curve algorithm based on its row and column numbers within the entire image range. For example, a tile located in the 3rd row and 5th column might have a Morton code of 001101 (binary representation).

[0024] Build a hybrid index: Construct a structured string with an index key in the format {spatial code}-{timestamp}-{attribute label}; It should be noted that {spatial encoding} serves as the spatial base of the index. A unique, one-dimensional Morton code is generated for each COG image tile using a space-filling curve (Z-curve). The Morton code preserves the proximity of tiles in two-dimensional space to the greatest extent possible; that is, tiles physically adjacent also have numerically similar Morton codes, laying a solid foundation for efficient range queries. {Timestamp} is the time dimension of the index, using a standard UTC timestamp to record the image acquisition time. Placing this after spatial encoding means the system can first identify a geographic area, and then within that area, all tiles are naturally sorted chronologically. {Attribute Labels} are the business dimension of the index, embedding the most critical and commonly used business attributes for filtering, such as cloud cover percentage, as labels. For example, an image with 10% cloud cover might have the attribute label CLOUD10.

[0025] Understandably, the composite key in a hybrid index can be efficiently stored in a distributed, sortable key-value database such as Apache HBase or Cassandra. Thus, a complex space-time-attribute filtering query can be transformed into an efficient scan of the database key range.

[0026] Metadata extraction and storage: Use the GDAL or Geotrellis library to parse COG files and extract key metadata, including spatial extent, projection information, number of bands, acquisition time, and cloud cover.

[0027] Associate these metadata entries with the generated hybrid index keys and store them in a separate column family in a metadata database (such as HBase / Cassandra) to lay the foundation for rapid data discovery and retrieval.

[0028] Geotrellis is a high-performance raster data processing library that enables rapid reading, writing, and processing of raster data, and supports various map geometry operations. Furthermore, Geotrellis supports rendering raster data as PNG format and saving raster file metadata in JSON format. By creating scalable, high-performance, distributed geospatial processing web services, Geotrellis can efficiently process massive datasets and leverages the advantages of multi-core architecture to perform parallel operations, thereby improving processing efficiency and accuracy.

[0029] S2. Distributed Data Organization: Based on a preprocessed hybrid index, target tiles are quickly located through spatial queries. Data blocks are accurately read from HDFS and spatially partitioned TileLayerRDDs are built in the Spark cluster. A persistent caching mechanism is also introduced to improve the access efficiency of high-frequency data, such as... Figure 3 As shown, it includes the following steps: S21. Distributed storage and data association: The pre-processed COG files are uploaded to HDFS for persistent storage. Geotrellis' catalog module creates a communication entry for each COG file, including the physical address and hybrid index key.

[0030] Understandably, the catalog module is used to record the storage path of each COG file on HDFS in the metadata database and associate it with the hybrid index key generated in the previous step, so that the abstract index key is successfully "anchored" to the specific physical data, thus building a connection bridge from logical query to physical file.

[0031] S22. Constructing TileLayerRDD: Locate AOI tiles using a hybrid index, read data after parsing the file path and tile offset, and construct TileLayerRDD in the Spark cluster. Specific steps include: For a specific AOI region, first query the metadata database and use the hybrid index to quickly locate the list of hybrid index keys corresponding to all tiles covering the current AOI region; For each key in the mixed index key list, Geotrellis queries the metadata to parse the COG file path and tile offset corresponding to each key, and then reads the tile data block through HadoopValueReader; The read tile data blocks are sent to the Spark cluster, and TileLayerRDD objects are constructed based on the spatial key information inherent in the tiles. Spark's partitioner assigns spatially adjacent tiles (with similar spatial key values) to the same compute node.

[0032] Understandably, these read tiles are organized into TileLayerRDD[SpatialKey] objects in Spark. The SpatialKey is essentially a wrapper around the tile's row and column numbers, and each partition of the RDD contains a series of spatially adjacent tiles, providing optimal data locality for subsequent parallel computation. This embodiment relies on Geotrellis to construct a distributed in-memory object TileLayerRDD dataset to achieve high-efficiency parallel computation. TileLayerRDD is a key-value pair RDD with metadata information, where the key represents a spatial index and the value represents a raster tile; it is the lowest-level data structure in the core of raster computation. The core raster computation is based on TileLayerRDD, providing various raster data operation operators, such as projection transformation and map algebra. Performing raster operations on different partitions of the TileLayerRDD enables distributed computation of massive amounts of raster data.

[0033] S23. Introduce a smart caching mechanism: For TileLayerRDDs that are frequently accessed and computed, call the rdd.persist method to cache them in the memory or SSD of the Spark cluster.

[0034] Understandably, by introducing a smart caching mechanism, the data flow for requests to the same data area will be truncated in the second step of "building TileLayerRDD", and the complete RDD will be retrieved directly from this memory or SSD cache, completely skipping the time-consuming "querying metadata -> precise reading" I / O path, thereby achieving a fast response.

[0035] Among them, rdd.persist(StorageLevel.MEMORY_AND_DISK_SER) is a core method in the Spark framework used to optimize computing performance. Its function is to persist (or cache) an RDD (Resilient Distributed Dataset) on multiple computing nodes in the cluster to avoid duplicate computations on the same dataset.

[0036] S3. Parallel computation of image data: Dynamically assemble and compute directed acyclic graphs based on algorithm repository templates, submit to a Spark cluster for distributed in-memory computation, and utilize Kubernetes to achieve real-time elastic scaling of computing resources to ensure processing performance. Figure 4 As shown, it includes the following steps: S31. Dynamic assembly of computational tasks: Configure the algorithm repository, encapsulate remote sensing indices such as NDVI, NDBI, and MNDWI into standard Geotrellis operators, and dynamically construct and execute directed acyclic graphs.

[0037] The algorithm repository is a scalable, persistent repository (e.g., a database table or configuration file) that defines each remote sensing index or process flow (such as NDVI, NDBI, reprojection, change detection) as a standardized, serializable computation template. Each template explicitly specifies: required inputs: for example, calculating NDVI requires "near-infrared band" and "red band"; computation logic: a description of the operation performed on the input data, such as (NIR - Red) / (NIR + Red); output specifications: the data type and range of the result, and parameters: such as small constants to prevent division by zero.

[0038] It should be noted that dynamically constructing the directed acyclic graph includes: Request parsing: When the microservice layer receives a request such as "Calculate AOI region NDVI", it first retrieves the calculation template corresponding to "NDVI" by name from the algorithm repository; Logical plan generation: The backend service parses the bands to be extracted from the TileLayerRDD based on the calculation template and generates a logical execution plan. This plan is a directed acyclic graph consisting of multiple operation steps, for example: [Read band 4] - [Read band 3] - [Execute NDVI algebra operation].

[0039] Physical plan binding: This binds the logical plan to a currently available TileLayerRDD containing concrete data. At this point, an executable, distributed directed acyclic graph (DAG) is formed within the Spark engine. This DAG precisely describes how data flows from the source through various transformation operations to ultimately produce the result.

[0040] In the fields of remote sensing, geographic information systems, and image processing, AOI (Area of ​​Interest) refers to a specific geographic area that a user focuses on and wants to manipulate and analyze. This area is typically defined by a regular geometric shape (such as a rectangle or circle) or an irregular polygon.

[0041] S32. Distributed Execution: Based on the directed acyclic graph (DAG), the Spark scheduler assigns each partition of the bound TileLayerRDD to a compute node. On each node, Geotrellis operators apply user-defined operations as closure functions to each tile of data within the partition. All these computations are executed in parallel in the node's memory, avoiding traditional disk I / O bottlenecks and achieving orders-of-magnitude speed improvements.

[0042] It's worth noting that the Spark engine performs a global analysis and optimization of the entire directed acyclic graph before execution. For example, it merges multiple consecutive, independent mapping operations into a single operation stage. This means that multiple computational steps can be completed in a single traversal, greatly reducing the generation of intermediate results and disk overflow, thus improving overall execution efficiency.

[0043] Spark is a general-purpose parallel computing framework that combines the advantages of Hadoop MapReduce with the ability to store intermediate job outputs in memory. This feature makes Spark more suitable than MapReduce for iterative MapReduce algorithms and tasks such as data storage, mining, analysis, and extraction. Using Spark's parallel computing framework, dimensionality-reduced key-value pairs can be stored, retrieved, analyzed, and processed in parallel.

[0044] S33. Resource Elastic Scheduling: Deploy the Spark computing cluster on the Kubernetes container orchestration platform, configure an elastic controller to continuously monitor the task queue length or resource load of the Spark driver behind the microservice gateway, and execute scheduling policies. The scheduling strategies include: When a backlog of tasks or a shortage of computing resources are detected, the controller automatically and quickly increases the number of Spark Executor Pod instances via the Kubernetes API to rapidly alleviate the computing pressure and ensure the real-time performance of the service.

[0045] When the system is idle, the controller will automatically reduce the number of redundant Executors and release idle computing resources to other services in the cluster, thereby optimizing costs.

[0046] Kubernetes is an open-source container orchestration platform. At its core is a declarative control system for automating the deployment, scaling, and management of containerized applications. It provides an application-oriented cluster management abstraction layer with containers as the basic scheduling unit, designed to automate production-level service operations.

[0047] S4. Remote Sensing Big Data Platform Service Release: This service encapsulates distributed computing capabilities into a RESTful API using a microservice architecture, allowing users to submit processing requests in real time. After dynamically completing calculations, projections, and caching, it instantly returns thematic maps and statistical results. Figure 5 As shown, it includes the following steps: S41. Create a microservice architecture and APIs: Break down the complex remote sensing processing system into a set of independent microservices based on functional modules. For example, create user management services, metadata query services, real-time computing services, and data rendering services respectively.

[0048] In front of all these microservices, deploy an API gateway as the sole access point for the entire system. Configure routing rules on this gateway to map different request paths to the corresponding backend microservices (for example, route all requests starting with / api / metadata / to the metadata query service).

[0049] At the same time, enable and configure authentication, access control, and request rate limiting policies on the gateway to ensure system security and stability.

[0050] S42. Service Chain Collaboration and Backpressure Mechanism: When the backend processing service receives a computation request forwarded by the API gateway, it activates the service chain for collaboration, including: Parsing and Querying: The processing service first parses the request parameters, then initiates a query to the metadata database. Utilizing a constructed hybrid index, it quickly locates the list of data tiles that need to be processed.

[0051] Task assembly and submission: The service dynamically assembles the computation process from the algorithm repository according to the requested algorithm, and then submits a distributed computation job to the Spark cluster.

[0052] At the same time, a backpressure mechanism is implemented for system self-protection, and the resource utilization (such as CPU and memory) and the length of the pending task queue of the Spark cluster are continuously monitored. When monitoring metrics show that cluster resources are nearing saturation, the system automatically triggers the backpressure mechanism: Reduce request reception rate: Limit the reception rate of new computing requests through the API gateway; Queuing or rejecting requests: Requests exceeding processing capacity are buffered by being directed to a message queue, or a "System busy" message is returned directly to prevent a flood of requests from overwhelming the computing cluster.

[0053] S43. Dynamic Output and Caching of Multiple Coordinate Systems: Reads the target coordinate system, calls the distributed reprojection operator to transform the TileLayerRDD, and generates results in the specified coordinate system; simultaneously, a caching mechanism is built to query / store calculation results via composite keys, including the following steps: Perform dynamic coordinate projection: Identify output coordinate system parameters: Read and parse the target coordinate system parameters specified in the user request; Triggering distributed reprojection computation: For requests that require coordinate system transformation, the Geotrellis reproject operator is called to initiate a distributed, parallel coordinate transformation operation on the TileLayerRDD that carries the computation results; Generate multi-coordinate system results: Produce new TileLayerRDD results that conform to the user-specified coordinate system, enabling instant conversion of the same data source to multiple standard map services.

[0054] Build and enable a projection result caching mechanism: Design cache key generation rules: Develop a composite key generation rule based on algorithm name, hash value of requested region spatial range, and target coordinate system to generate a globally unique cache identifier for each computation request; Implement cache query and write: Before returning the final result, first query the cache database using the generated cache key; if it exists, return the cached result directly; if it does not exist, execute the complete calculation and projection process, and after the process is completed, associate the final result with the cache key and store it in the cache database.

[0055] Configure cache management strategies: Configure cache lifetime and eviction policies to ensure the freshness of cached data and the effective use of storage space.

[0056] In Geotrellis, `reproject` is a distributed, parallel-execution spatial reference system transformation operator. Its core function is to transform a `TileLayerRDD` (i.e., a raster dataset distributed across a Spark cluster) from its current coordinate system (e.g., the original WGS84 latitude and longitude coordinate system of an image) to another specified target coordinate system (e.g., the UTM projection used for engineering surveying or the Web Mercator coordinate system commonly used for web maps), and generate a new `TileLayerRDD` located in the target coordinate system.

[0057] S44. Result Return: Render the final TileLayerRDD (whether computed in real time or retrieved from the cache) into a PNG format thematic map image; at the same time, quickly perform statistical operations on the distributed cluster (such as calculating the average NDVI value, maximum value, minimum value, etc. of the entire region).

[0058] The backend service packages the generated thematic map images and statistical results and returns them to the frontend browser via API. After receiving the results, the frontend application (such as a webpage built on OpenLayers or Leaflet) clearly displays the thematic map to the user and presents the statistical values ​​in the form of charts or numbers.

[0059] This embodiment also provides a remote sensing image data processing system based on the Spark architecture, such as... Figure 6 As shown, it includes a data preprocessing and index building module, a distributed storage and data organization module, a parallel computing and task scheduling module, a service publishing and interface management module, a front-end interaction and visualization module, and a system monitoring and resource management module.

[0060] The data preprocessing and indexing module is responsible for preprocessing the original remote sensing image data, constructing a multi-dimensional hybrid index, and extracting and storing the image's metadata (such as spatial range, projection, band, time, cloud cover, etc.) into the metadata database, providing a data foundation for subsequent rapid retrieval and positioning.

[0061] It should be noted that the data preprocessing and index building module uses GDAL to batch convert raw GeoTIFF and other data into cloud-optimized GeoTIFF format, giving it tiled and pyramidal structure to support efficient partial reading. Subsequently, Z-curves are used to spatially encode each tile, generating a composite index key that integrates spatial location, timestamp, and key attributes (such as cloud cover). At the same time, metadata such as spatial extent, projection, and band of the image is automatically extracted and stored in a distributed database after being associated with the index key, thereby enabling rapid data discovery and accurate positioning.

[0062] The distributed storage and data organization module is used to store the preprocessed COG files into the HDFS distributed file system, establish a mapping relationship between index keys and physical file paths through the metadata catalog, quickly locate and construct TileLayerRDD based on the AOI area requested by the user, and introduce an intelligent caching mechanism to persist frequently accessed RDDs in memory or SSD to improve the efficiency of repeated access.

[0063] It should be noted that this module aims to provide high-throughput, low-latency data organization and access services for parallel computing in massive data environments, and optimizes the response speed of high-frequency requests through a caching mechanism. Processed COG files are persistently stored in HDFS. When a user request is received, the system uses a pre-built hybrid index to quickly locate the target tile in the metadata database, resolves its physical storage path and offset, and then loads the data into the Spark cluster through an efficient reader, organizing it into spatially partitioned TileLayerRDDs to optimize data locality. Simultaneously, frequent RDDs are persisted and cached in memory or on disk, thus achieving near-instantaneous response during repetitive computations.

[0064] The parallel computing and task scheduling module provides an algorithm repository with pre-set standard operators, constructs and submits a directed acyclic graph to the Spark cluster to achieve distributed in-memory computing, and utilizes Kubernetes to achieve elastic scaling of computing resources, dynamically adjusting the number of Executors according to the task load to ensure real-time performance and resource utilization.

[0065] It should be noted that this module is the computational core of the system, responsible for transforming complex remote sensing analysis tasks into distributed, in-memory parallel computing and implementing elastic management of computing resources. It utilizes a pre-built algorithm repository with standard operators such as NDVI. Upon receiving a computation request, it dynamically assembles the corresponding directed acyclic graph (DAG) and binds it to the data RDD before submitting it to the Spark cluster. The Spark scheduler distributes the computation tasks to each node, where the Geotrellis operator processes the tile data within each partition in parallel. The entire computing cluster is deployed on Kubernetes, enabling automatic scaling of computing resources based on real-time load to ensure both computational performance and cost-effectiveness.

[0066] An Executor is a worker process in an Apache Spark application, specifically responsible for executing computational tasks on a particular node in the cluster and managing the data for those tasks on that node. It is the execution endpoint of Spark's distributed computing model and the entity that actually performs the data processing.

[0067] The service publishing and interface management module adopts a microservice architecture. It handles authentication, authorization, and routing uniformly through an API gateway. The backend service chain is responsible for parsing requests, locating data, and dynamically submitting computation jobs. It automatically limits or queues requests by monitoring cluster load. At the same time, the computation results support dynamic coordinate reprojection and generate a cache key together with the request parameters. For duplicate requests, the cached results are returned directly.

[0068] Understandably, this module serves as the unified portal for the system to the outside world, encapsulating the underlying capabilities into stable and secure RESTful APIs, and providing a service governance mechanism to ensure high system availability.

[0069] The front-end interaction and visualization module is used to realize the interaction of geographic information, the initiation of analysis tasks, and the dynamic visualization of calculation results by building a web user interface.

[0070] It should be noted that this module is built on a web application based on libraries such as Vue.js and OpenLayers / Leaflet. Users can browse and draw analysis areas on the front-end map. The front-end application submits calculation requests by calling the back-end API and receives the returned thematic maps (such as PNG format) and statistical results (such as JSON data), which are then rendered and displayed in an integrated manner in the map interface and chart components.

[0071] The system monitoring and resource management module is used to continuously collect various indicators from infrastructure to application services by deploying a comprehensive monitoring agent, perform unified management of user permissions, access logs and resource quotas, and trigger alarms when anomalies or resource bottlenecks are detected, providing decision support for system optimization and stable operation and maintenance.

[0072] Understandably, this module is the backbone of the system's operation and maintenance, responsible for monitoring, managing, and maintaining the entire process to ensure the platform's continuous, stable, and efficient operation.

[0073] This invention firstly achieves rapid retrieval and region-of-interest localization of massive image data through the combined use of Cloud Optimized GeoTIFF (COG) and a spatial-temporal-attribute hybrid index, significantly improving data access efficiency. Secondly, a distributed in-memory computing core built on Spark and Geotrellis, combined with elastic resource scheduling implemented by Kubernetes, enables rapid response to complex remote sensing index calculations (such as NDVI), realizing a shift from "data download and post-processing" to "real-time cloud computing." Finally, through microservice encapsulation and dynamic computation chain assembly, a standardized and scalable RESTful API is provided to support high-concurrency access by multiple users. Backpressure and caching mechanisms ensure service stability and high performance, ultimately providing users with an integrated, real-time online service experience from data browsing and on-demand analysis to result visualization.

[0074] This invention first reconstructs and intelligently indexes the raw data using a "cloud-native" approach, uniformly converting multi-source remote sensing imagery into the COG format that supports HTTP range requests. It then constructs a multi-dimensional hybrid index that integrates spatial location, temporal, and business attributes, addressing the fundamental problems of slow data reading and inaccurate data retrieval. Next, a distributed data organization and parallel computing architecture is designed. HDFS is used for persistent data storage, and Spark is used to organize the data into TileLayerRDDs with spatial locality. Combined with high-order raster operators provided by Geotrellis, a distributed in-memory computing engine capable of parallel processing of massive amounts of tiles is built. Simultaneously, resource elasticity and dynamic computation assembly mechanisms are introduced. Based on Kubernetes, computing resources are scaled on demand, and an "algorithm repository" is used to template analytical models, supporting the dynamic assembly and execution of directed acyclic graphs based on user requests, achieving flexible computation with "one database, multiple functions, and on-demand assembly." Ultimately, the capabilities are encapsulated and delivered through microservices and intelligent governance. The powerful underlying distributed computing capabilities are exposed as standardized services through the API gateway, and functions such as backpressure, multi-level caching, and dynamic projection are integrated to ensure service resilience in high-concurrency scenarios, providing users with an end-to-end real-time remote sensing computing solution.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A remote sensing image data processing method based on Spark architecture, characterized in that: Includes the following steps: Data preprocessing and index building: The collected raw files are converted into COG format, a hybrid index is built using spatial dimensionality reduction technology, metadata is extracted and a metadata database is established; Distributed data organization: Based on hybrid indexes, target tiles are quickly located through spatial queries, data blocks are read and spatially partitioned TileLayerRDDs are built in the Spark cluster, while a persistent caching mechanism is introduced; Parallel computation of image data: Dynamically assemble and compute a directed acyclic graph based on the algorithm repository template, submit it to the Spark cluster for distributed in-memory computation, and realize real-time elastic scaling of computing resources through Kubernetes; Remote sensing big data platform service release: Through microservice architecture, distributed computing is encapsulated as a RESTful API, which supports users to submit processing requests in real time, dynamically completes calculations, projections and caching, and immediately returns thematic maps and statistical results.

2. The remote sensing image data processing method based on Spark architecture according to claim 1, characterized in that: The data preprocessing and index construction includes the following steps: Standardized data import: Use the GDAL command-line tool or write automated scripts to batch convert the collected raw format data into COG files; Spatial dimensionality reduction index: Morton codes are generated by encoding COG tiles using Z-curves, a hybrid index is constructed, and metadata is extracted and stored using GDAL or Geotrellis.

3. The remote sensing image data processing method based on Spark architecture according to claim 2, characterized in that: The spatial dimensionality reduction index includes the following steps: Z-curve encoding: For each tile in the COG file, a unique, one-dimensional Morton code is calculated using the Z-curve algorithm based on the row and column numbers within the entire image range. Build a hybrid index: Construct a structured string with an index key in the format {spatial code}-{timestamp}-{attribute label}; Metadata extraction and storage: Use the GDAL or Geotrellis library to parse COG files and extract key metadata, including spatial extent, projection information, number of bands, acquisition time, and cloud cover. The extracted metadata is associated with the generated hybrid index key and stored in a separate column family of metadata.

4. The remote sensing image data processing method based on Spark architecture according to claim 1, characterized in that: The distributed data organization includes the following steps: Distributed storage and data association: The pre-processed COG files are uploaded to HDFS for persistent storage, and a communication entry is created for each COG file, including the physical address and hybrid index key; Constructing TileLayerRDD: Locate AOI tiles using a hybrid index, read data after parsing the file path and tile offset, and construct TileLayerRDD in the Spark cluster; Introducing a smart caching mechanism: For TileLayerRDDs that are frequently accessed and computed, the rdd.persist method is called to cache them in the memory or SSD of the Spark cluster.

5. The remote sensing image data processing method based on Spark architecture according to claim 4, characterized in that: The construction of TileLayerRDD includes the following steps: For a specific AOI region, first query the metadata database and use the hybrid index to quickly locate the list of hybrid index keys corresponding to all tiles covering the current AOI region; For each key in the mixed index key list, Geotrellis queries the metadata to parse the COG file path and tile offset corresponding to each key, and then reads the tile data block through HadoopValueReader; The read tile data blocks are sent to the Spark cluster, and TileLayerRDD objects are constructed based on the spatial key information inherent in the tiles. Spark's partitioner allocates spatially adjacent tiles to the same compute node.

6. The remote sensing image data processing method based on Spark architecture according to claim 1, characterized in that: The image data is processed in parallel, including the following steps: Dynamic assembly of computational tasks: Configure the algorithm repository, encapsulate NDVI, NDBI and MNDWI remote sensing indices into standard Geotrellis operators, and dynamically construct and execute directed acyclic graphs. Distributed execution: Based on the execution directed acyclic graph, the Spark scheduler assigns each partition of the bound TileLayerRDD to a compute node. On each node, Geotrellis operators apply user-defined operations as closure functions to each tile of data within the partition. Resource elastic scheduling: Deploy the Spark computing cluster on the Kubernetes container orchestration platform, configure an elastic controller to continuously monitor the task queue length or resource load of the Spark driver behind the microservice gateway, and execute scheduling policies.

7. The remote sensing image data processing method based on Spark architecture according to claim 6, characterized in that: The dynamic construction of the directed acyclic graph includes: Request parsing: When the microservice layer receives a request, it first retrieves the corresponding calculation template by name from the algorithm repository; Logical plan generation: The backend service parses out the bands that need to be extracted from TileLayerRDD based on the calculation template and generates a logical execution plan; Physical plan binding: Binds the logical plan to a currently ready TileLayerRDD that contains concrete data; In the aforementioned elastic resource scheduling, the scheduling strategies include: When a backlog of tasks or a shortage of computing resources is detected, the controller automatically and quickly increases the number of Spark Executor Pod instances via the Kubernetes API. When the system is idle, the controller will automatically reduce the number of redundant Executors and release the idle computing resources to other services in the cluster.

8. The remote sensing image data processing method based on Spark architecture according to claim 1, characterized in that: The release of the remote sensing big data platform service includes the following steps: Create a microservice architecture and API: break down the complex remote sensing processing system into a set of independent microservices according to functional modules; deploy an API gateway in front of all microservices and configure routing rules to map different request paths to the corresponding backend microservices; at the same time, enable and configure authentication, access control and request rate limiting policies on the gateway. Service chain collaboration and backpressure mechanism: When the backend processing service receives a computation request forwarded by the API gateway, it activates the service chain for collaboration, including: Parsing and Querying: The processing service first parses the request parameters, then initiates a query to the metadata database, and uses the constructed hybrid index to quickly locate the list of data tiles that need to be processed; Task assembly and submission: The service dynamically assembles the computation process from the algorithm repository according to the requested algorithm, and then submits a distributed computing job to the Spark cluster; Implement a backpressure mechanism for system self-protection, continuously monitoring the resource utilization and pending task queue length of the Spark cluster; when monitoring metrics show that cluster resources have reached the saturation threshold, the backpressure mechanism is automatically triggered. Reduce request reception rate: Limit the reception rate of new computing requests through the API gateway; Queuing or rejecting requests: Requests exceeding processing capacity are buffered by being directed to a message queue, or a "system busy" message is returned directly to prevent a flood of requests from overwhelming the computing cluster; Dynamic output and caching across multiple coordinate systems: The system reads and resolves the target coordinate system, calls the distributed reprojection operator to transform the TileLayerRDD, and generates results in the specified coordinate system. Simultaneously, a caching mechanism is built to query / store computation results via composite keys, including the following steps: Perform dynamic coordinate projection: Identify output coordinate system parameters: Read and parse the target coordinate system parameters specified in the user request; Triggering distributed reprojection computation: For requests that require coordinate system transformation, the Geotrellis reproject operator is called to initiate a distributed, parallel coordinate transformation operation on the TileLayerRDD that carries the computation results; Generate multi-coordinate system results: Produce new TileLayerRDD results that conform to the user-specified coordinate system, enabling instant conversion of the same data source to multiple standard map services; Build and enable a projection result caching mechanism: Design cache key generation rules: Develop a composite key generation rule based on algorithm name, hash value of requested region spatial range, and target coordinate system to generate a globally unique cache identifier for each computation request; Implement cache query and write: Before returning the final result, first query the cache database using the generated cache key; if it exists, return the cached result directly; if it does not exist, execute the complete calculation and projection process, and after the process is completed, associate the final result with the cache key and store it in the cache database; Configure cache management strategies: Configure cache lifetime and eviction policies to ensure the freshness of cached data and the effective use of storage space; The result returned is: the final TileLayerRDD is rendered into a PNG format thematic map image; and statistical operations are performed quickly on the distributed cluster. The backend service packages the generated thematic map images and statistical results, and returns them to the frontend browser via API. After receiving the results, the frontend application clearly displays the thematic map to the user.

9. A remote sensing image data processing system based on Spark architecture, executing the remote sensing image data processing method based on Spark architecture as described in any one of claims 1-8, characterized in that: It includes modules for data preprocessing and index building, distributed storage and data organization, parallel computing and task scheduling, service publishing and interface management, front-end interaction and visualization, and system monitoring and resource management. The data preprocessing and index building module is responsible for preprocessing the original remote sensing image data, building a multi-dimensional hybrid index, and extracting and storing the image metadata into the metadata database to provide a data foundation for subsequent rapid retrieval and positioning. The distributed storage and data organization module is used to store the preprocessed COG files into the HDFS distributed file, establish a mapping relationship between index keys and physical file paths through the metadata directory, quickly locate and construct TileLayerRDD according to the AOI area requested by the user, and introduce an intelligent caching mechanism to persist frequently accessed RDDs in memory or SSD. The parallel computing and task scheduling module is used to provide an algorithm repository with pre-set standard operators, build and submit a directed acyclic graph to the Spark cluster to realize distributed in-memory computing, use Kubernetes to realize elastic scaling of computing resources, and dynamically adjust the number of Executors according to the task load.

10. A remote sensing image data processing system based on Spark architecture according to claim 9, characterized in that: The service publishing and interface management module adopts a microservice architecture. It uses an API gateway to uniformly handle authentication, authorization and routing. The backend service chain is responsible for parsing requests, locating data and dynamically submitting computing jobs. It automatically limits or queues requests by monitoring the cluster load. Meanwhile, the calculation results support dynamic coordinate reprojection and are used together with the request parameters to generate a cache key, and the cached results are returned directly for repeated requests. The front-end interaction and visualization module is used to realize the interaction of geographic information, the initiation of analysis tasks, and the dynamic visualization of calculation results by building a web user interface; The system monitoring and resource management module is used to continuously collect various indicators from infrastructure to application services by deploying a comprehensive monitoring agent, perform unified management of user permissions, access logs and resource quotas, and trigger alarms when anomalies or resource bottlenecks are detected.

Images