A system and method for resampling raster data in a high concurrency scenario

Abstract

The present application relates to the field of grid data application, and particularly to a grid data resampling method and system in a high concurrency scenario, the system comprising: an extensible sampling module, an extensible cache computing module and a storage area, the storage module storing grid data, pyramid data, coordinate system data and index table data; the extensible cache computing module performs pyramid processing on newly uploaded grid data to obtain pyramid data; and a plurality of extensible cache computing modules work in parallel; the extensible sampling module comprises a plurality of independently running modules, which are used to respond to the demand for resampling, resample the pyramid data and feed back to the user. In the high concurrency scenario, distributed cache storage technology and distributed resampling technology are used to reduce the complexity of system design and optimize the efficiency of client data request.

Description

Technical Field

[0001] This invention relates to the field of raster data applications, and in particular to a raster data resampling system and method in high-concurrency scenarios. Background Technology

[0002] Raster data, a commonly used data format in Geographic Information Systems (GIS), is a specific storage scheme for spatial information by dividing space into regular grids, each grid representing a cell, and each cell having specific attribute information. Commonly used raster data formats include TIFF, GIF, and JPEG.

[0003] With the development of drone technology, high-definition orthophoto technology has rapidly emerged. As a storage method for high-definition orthophotos, raster data requires the storage of increasingly more information and has a larger data volume. Due to limitations in computer memory, the problem of raster data being unopenable and unviewable has arisen. To solve this problem, image pyramid technology is generally used to store raster data by copying it at progressively lower resolutions. When needed, a resolution similar to the display area is selected. This solution requires only a small amount of querying and calculation to solve the image display problem, greatly reducing the memory dependence and time consumption when applying large volumes of raster data.

[0004] With the rapid development of the internet, high concurrency has become a crucial factor in system architecture design. It typically refers to a system handling multiple requests simultaneously. Key metrics for evaluating high concurrency include response time, throughput, queries per second (MS / MS), and concurrent users. In recent years, with the development of cloud technology, more raster data applications have evolved from desktop standalone applications to system-level applications with cloud-edge-device architectures. Currently, mainstream GIS products such as ArcGIS and SuperMap process raster data into tile data, store it in a data warehouse, and then download and assemble the tile data based on front-end requirements to form a complete raster dataset. While this approach is adequate for low-concurrency access, it leads to computational and retrieval bottlenecks for high-concurrency access.

[0005] Traditional pyramid processing of large volumes of raster data has performance limitations, limiting its application to single-machine computers. Furthermore, the processed pyramid data is typically stored on a computer's hard drive, requiring image resampling for each specified area, which wastes a significant amount of computation and search time.

[0006] Traditional service-oriented raster data slicing solutions typically involve slicing on the server side, storing the slices on the hard drive, and then having the client access the data according to a protocol. Each time, real-time calculations are performed to obtain the correct slice data. This approach can lead to computational and retrieval bottlenecks during concurrent access, making it unable to meet the explosive demand from clients. Summary of the Invention

[0007] To address the issues of computational and retrieval bottlenecks caused by high-concurrency access, the waste of significant computation and search time in traditional pyramid processing of large volumes of raster data, and the inability of traditional service-oriented raster data slicing solutions to meet the explosive demand from clients, this invention establishes a distributed architecture and process that fully utilizes computational and storage resources. It also proposes a raster data resampling system and method for high-concurrency scenarios to meet the application requirements of raster data in such scenarios.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0009] A raster data resampling system for high-concurrency scenarios includes a scalable sampling module, a scalable cache computing module, and a storage area.

[0010] The storage module stores raster data, pyramid data, coordinate system data, and index table data;

[0011] The scalable cache computing module performs pyramid processing on the newly uploaded raster data to obtain pyramid data; and multiple scalable cache computing modules work in parallel.

[0012] The scalable sampling module includes multiple independently operating modules that respond to resampling requests, resample the pyramid data, and provide feedback to the user.

[0013] As a preferred embodiment, in the storage module, there is a correspondence between raster data and pyramid data, and between coordinate system data and raster data. The index table is a relational data table that stores indexes of raster data, pyramid data, and coordinate system data.

[0014] As a preferred embodiment, the storage module includes scalable NAS and MySQL storage record tables.

[0015] Raster data and pyramid data are stored in a scalable NAS, and the access method for the scalable NAS is stored in the MySQL storage record table; coordinate system data and index table data are recorded in the MySQL storage record table.

[0016] As a preferred embodiment, if the index table data lacks raster data, the scalable cache calculation module automatically generates corresponding pyramid data for the raster data and stores the corresponding pyramid data in the storage module, specifically including the following steps:

[0017] New raster data is uploaded to the storage module;

[0018] The newly uploaded raster data is cached and calculated, and then available computing resources are located based on the cache calculation results.

[0019] Idle computing resources initiate cache generation logic. On the one hand, the original image files required for cache generation are obtained from the storage module. On the other hand, the pyramid data is generated layer by layer through the initialization configuration parameters set by the system.

[0020] The newly generated pyramid data is stored in the storage module, and an index table is generated.

[0021] As a preferred embodiment, the scalable sampling module is used to retrieve the original raster data, pyramid data, and coordinate system data, and, in conjunction with the parameters requested by the user, resample the original raster data, return the resampling results to the user, store the resampling results in the cache module, and establish a corresponding retrieval index.

[0022] As a preferred embodiment, a storage lock is also included. The storage lock is used to create a lock information in the index table data for each access to scalable resampling. The lock is kept closed during the access and released only when the access ends, thereby ensuring that data access is not conflicting.

[0023] Based on the same concept, a method for resampling raster data in high-concurrency scenarios is also proposed. A raster data resampling system as described above in any of the above-mentioned scenarios is constructed, and the following steps are performed:

[0024] Step 1: Upload the raster data to the server and generate an image pyramid;

[0025] Step two: The client initiates a request to obtain image data of the specified area and level;

[0026] Step 3: Check if the resampled data exists in the cache of the storage module. If it exists, return the search result directly. If it does not exist, proceed to step 4.

[0027] Step 4: Resample the pyramid data according to the specified area and level, generate sampling results, and store them in the cache.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] The method of this invention enables a service-oriented application specifically for raster pyramid data. In high-concurrency scenarios, this technology employs distributed caching and distributed resampling techniques, which reduces the complexity of system design and optimizes the efficiency of client data requests. Attached image description:

[0030] Figure 1 This is a system architecture diagram of a raster data resampling system in a high-concurrency scenario according to Embodiment 1 of the present invention;

[0031] Figure 2 This is a structural diagram of the storage module in Embodiment 1 of the present invention;

[0032] Figure 3 This is a flowchart illustrating the generation of pyramid data after raster data is uploaded to the storage center in Embodiment 1 of the present invention.

[0033] Figure 4 This is a diagram illustrating the specific extension method for the cache calculation module in Embodiment 1 of the present invention;

[0034] Figure 5 This is a flowchart of a method for resampling raster data in a high-concurrency scenario according to Embodiment 1 of the present invention. Detailed Implementation

[0035] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0036] Example 1

[0037] An architecture diagram of a raster data resampling system in high-concurrency scenarios is shown below. Figure 1 As shown, the system includes a scalable sampling module, a scalable cache computing module, and a storage area. The storage module primarily uses hard disk storage or cloud storage to store the original raster data, coordinate system data, index table data, and pyramid data. The scalable cache computing module performs pyramid processing on newly uploaded raster data. This part has multiple computing modules working in parallel to ensure the fastest possible completion of pyramid processing. The scalable resampling module also has multiple independently running modules. Its key function is to respond to user resampling requests, resample from the pyramid, obtain the user's target data, and return it to the user.

[0038] Storage modules typically employ hard disk file systems or cloud databases for storage, storing both structured and unstructured data. Data is primarily categorized into raster data, pyramid data, coordinate system data, and indexed table data. A diagram of the storage module's structure is shown below. Figure 2 As shown.

[0039] Raster data refers to raw data, such as TIFF, GIF, or JPEG images. These data are generally characterized by large volume, fixed format, and low frequency of use. In order to store these data more reasonably, the system has established a dedicated raster data warehouse and developed basic data comparison, search, insertion, and deletion functions for the data warehouse.

[0040] The pyramid repository stores pyramid data generated from raster data. This data is generally used in conjunction with raster data. Therefore, when storing this data, the one-to-one binding relationship between the pyramid repository data and the raster data is taken into account. The data formats stored in the data warehouse include, but are not limited to, *over and *.rrd.

[0041] The coordinate system repository stores the coordinate system information of the raster data. Key positional information such as the scaling ratio and rotation angle of the raster data are identified by this file. There is a one-to-one correspondence between the information in this repository and the original raster data. Therefore, the system also stores this one-to-one correspondence. The data stored in the coordinate system repository includes, but is not limited to, file formats such as *.jgw and *.prj.

[0042] The index table, as the key table for retrieving raw data, is the starting point for the storage module. External programs using the storage module should first search the index table. All information and information relationships are stored in the index table. The index table is a relational data table that mainly stores all raster data, pyramid data, coordinate system data indexes in the system, as well as the information and relationships of these data.

[0043] The main challenges in designing the storage area were storage expansion and fast retrieval. To address these issues, the system design employed corresponding storage strategies for different data types. Raster and pyramid data, primarily unstructured, are characterized by large single-file volumes, difficulty in structuring, and limited copying. Such data is unsuitable for database storage due to significantly reduced access efficiency. Furthermore, traditional hard drive storage faces challenges in cross-server access due to its limited scalability to other business servers. NAS storage effectively solves these problems, offering both suitability for unstructured data storage and cross-server access over a network. This invention extends NAS storage management by adding storage records to MySQL, retaining both the advantages of NAS for storing large volumes of unstructured data and the ease of integration of structured data into business systems, thus enabling the storage of raster and pyramid data. Coordinate data exhibits diversity, including PRJ data, XML data, and some unknown extended data. Therefore, the storage and management of coordinate data in this invention employs the same management scheme as for raster and pyramid data. To ensure the stable operation of the system, a dedicated monitoring module was designed in the invention. The monitoring module can perform real-time health diagnosis, abnormal warnings, and automatic health status repair of the storage.

[0044] The scalable cache computing module's primary task is to generate pyramid data. For user-submitted raw data, it can collect relevant data, invoke computing processes, and generate cache results immediately. For raster data lacking pyramid data, the system provides an automatic pyramid generation program. After a user uploads raster data, the computing module searches the index table; if it finds that the raster data lacks pyramid data, it will automatically generate the corresponding pyramid data. The flowchart for generating pyramid data after raster data is uploaded to the storage center is shown below. Figure 3 As shown.

[0045] After the raster data is uploaded to the storage center, the upload program sends a new task message to the task center, notifying the task center to perform cache calculations on the newly uploaded raster data. After the task center successfully completes the task, it calls the idle computing resource finder in the computing center to search for idle computing resources. Once an idle resource is identified, the idle computing resource starts the cache generation logic. On the one hand, it obtains the newly uploaded raster data required for cache generation from the storage center. On the other hand, it starts the layer-by-layer generation of pyramid data through the system's initialization configuration parameters. Finally, the generated data is stored in the storage center again, and an index table is generated for subsequent service calls.

[0046] During cache generation, basic operations such as task interruption and restart are supported. Tasks in the task center are also controlled directly through simple commands without the user's awareness. In the computing center, a single computing resource can only process a single image data at a time. Therefore, when the task center is overloaded, its computing resources become strained. In this situation, administrators can automatically expand computing resources through simple configuration. Specific expansion methods for the cache computing module are as follows... Figure 4 As shown. When computing center resources are strained, a new computer is allocated, ensuring that the new computer can access the computing resource pool and storage pool. The computing module application is installed and run on the new computer. After the program runs, the computing resource pool has an additional idle computing module. This module will actively access the tasks to be processed, obtain the task ID and corresponding information, lock the task, retrieve the image data to be processed from the storage pool, perform pyramid data processing, and store the processing results in the storage pool after processing is complete.

[0047] The scalable resampling module, based on the raster data area and sampling rate required by the user, first searches the cache module. If the requested data is not found in the cache module, the resampling module regenerates it. The resampling module retrieves the original raster data, pyramid data, and coordinate system data, combines them with user-requested parameters (such as image windows), resamples the original raster, returns the resampling results to the user, and stores them in the cache module, establishing a corresponding retrieval index. The request processing module acts as an index for the calculation module. All client requests first enter the request processing module, which categorizes the request type, determining whether it's login authentication, cache management, or data warehouse management, before assigning the request to the designated module for processing.

[0048] The resampling module is a scalable sampling module that, based on the user's resampling request, searches for available sampling modules, and then uses these modules to retrieve cached data or resample existing data.

[0049] (1) Raw image data for existing pyramid data.

[0050] For the existing pyramid data's raw image data, the resampling module directly accesses and uses the pyramid data.

[0051] (2) For image data for which pyramid data has not yet been generated.

[0052] First, check if there are any ongoing tasks in the task center. If so, wait for the task to complete before sampling. If there are no tasks with current image data in the task center, create a new task, generate pyramid data, and then perform sampling.

[0053] Since the extensible module has multiple extensible service plugins, there may be conflicting access issues when accessing data in the storage area. To address this issue, this invention creates a storage lock.

[0054] In the index data, a lock is created for each access in the scalable resampling. The lock is kept closed during the access and released only when the access ends, thus ensuring that data access is not conflicting.

[0055] Example 2

[0056] Based on the raster data resampling system in high-concurrency scenarios in Example 1, a method for raster data resampling in high-concurrency scenarios is also proposed. The flowchart of the method is as follows: Figure 5 As shown, it includes the following steps:

[0057] Step 1: Upload the raster data to the server and generate an image pyramid.

[0058] Step two: The client initiates a request to obtain image data of the specified area and level.

[0059] Step 3: Check if resampled data exists in the cache. If it exists, return the search result directly. If it does not exist, proceed to step 4.

[0060] Step 4: Resample the pyramid data according to the specified area and level, generate sampling results, and store them in the cache.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A raster data resampling system for high-concurrency scenarios, characterized in that, It includes a scalable sampling module, a scalable cache computing module, and a storage module. The storage module stores raster data, pyramid data, coordinate system data, and index table data; The scalable cache computing module performs pyramid processing on the newly uploaded raster data to obtain pyramid data; and multiple scalable cache computing modules work in parallel. The scalable sampling module includes multiple independently operating modules for responding to resampling needs, resampling the pyramid data, and providing feedback to the user. The scalable cache computing module generates pyramid data specifically through the following steps: After the raster data is uploaded to the storage module, the upload program sends a message to the task center to create a new task, notifying the task center to perform cache calculations on the newly uploaded raster data. After the task center caches the raster data, it calls the idle computing resource finder in the computing center to search for idle computing resources. Once an idle resource is identified, the idle computing resource starts the cache generation logic. On the one hand, it obtains the newly uploaded raster data required for cache generation from the storage center. On the other hand, it starts the layer-by-layer generation of pyramid data through the system's initialization configuration parameters. Finally, the generated pyramid data is stored in the storage module again, and an index table is generated for subsequent service calls. When computing center resources are strained during the generation of pyramid data by the scalable cache computing module, a new computer is allocated, ensuring that the new computer can access the computing resource pool and storage module. The scalable cache computing module application is installed and run on the new computer. After the program runs, the computing resource pool will have an additional scalable cache computing module in an idle state. The added scalable cache computing module actively accesses the tasks to be processed, obtains the task ID and corresponding information, locks the tasks to be processed, retrieves the image data to be processed from the storage module, performs the pyramid data processing work, and stores the processing results in the storage module after processing is completed. The scalable sampling module is used to retrieve the original raster data, pyramid data, and coordinate system data, and, in combination with the parameters requested by the user, resample the original raster data, return the resampling results to the user, store the resampling results in the scalable cache calculation module, and establish a corresponding retrieval index. It also includes a storage lock, which is used to create a storage lock information for each access in the index table data for scalable resampling. The storage lock is kept closed during the access and released only when the access ends, thereby ensuring that the data access is not conflicting. The storage module includes scalable NAS and MySQL storage record tables. Raster and pyramid data are stored in a scalable NAS and stored in a MySQL storage record table. Scalable NAS access methods; coordinate system data and index table data are recorded in MySQL storage record tables.

2. The raster data resampling system in a high-concurrency scenario as described in claim 1, characterized in that, In the storage module, there is a correspondence between raster data and pyramid data, and between coordinate system data and raster data. The index table is a relational data table that stores indexes of raster data, pyramid data, and coordinate system data.

3. The raster data resampling system in a high-concurrency scenario as described in claim 1, characterized in that, If the index table data lacks raster data, the scalable cache computing module automatically generates corresponding pyramid data for the raster data and stores the corresponding pyramid data in the storage module, specifically including the following steps: New raster data is uploaded to the storage module; The newly uploaded raster data is cached and calculated, and then available computing resources are located based on the cache calculation results. Idle computing resources initiate cache generation logic. On the one hand, the original image files required for cache generation are obtained from the storage module. On the other hand, the pyramid data is generated layer by layer through the initialization configuration parameters set by the system. The newly generated pyramid data is stored in the storage module, and an index table is generated.

4. A method for resampling raster data in high-concurrency scenarios, characterized in that, Construct a raster data resampling system as described in any one of claims 1-3 under high concurrency scenarios, and perform the following steps: Step 1: Upload the raster data to the server and generate an image pyramid; Step two: The client initiates a request to obtain image data of the specified area and level; Step 3: Check if the resampled data exists in the cache of the storage module. If it exists, return the search result directly. If it does not exist, proceed to step 4. Step 4: Resample the pyramid data according to the specified area and level, generate sampling results, and store them in the cache.

Images