Method, device and system for processing spatial data

By identifying and merging duplicate tiles in spatial data, the problem of low storage and bandwidth efficiency in the prior art is solved, efficient spatial data processing and maintenance is realized, and geographic map tiles are supported on-demand rendering, suitable for vehicle navigation systems. etc.

CN118946907APending Publication Date: 2024-11-12GRABTAXI HOLDINGS PTE LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380030124.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-11
Filing Date
2023-04-13
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is difficult to effectively process exponentially growing spatial data sets, resulting in storage and bandwidth problems, and there is duplicate data storage, affecting processing and maintenance efficiency.

Method used

By receiving spatial data files, generate the associated tiles set, compare tiles identifiers, identify and merge duplicate tiles, generate duplicate indicators, and package the merged tiles into file format and store them in a specified directory , used to create a self-hosted satellite map tiles service for users to access on demand.

Benefits of technology

Reduces duplicate data storage, improves processing efficiency, supports rendering of geomap tiles on demand, is suitable for applications such as vehicle navigation systems, and improves the storage and maintenance capabilities of spatial data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946907A_ABST
    Figure CN118946907A_ABST
Patent Text Reader

Abstract

Some aspects relate to a method for processing spatial data, the method comprising the steps of: receiving a spatial data file from a spatial data source; generating a first tile set associated with the spatial data file, wherein the first tile set comprises a plurality of tiles and corresponding identifiers; comparing the identifier of each of the plurality of tiles in the first set of tiles with the identifier of each tile in the second set of tiles to identify at least one repetition; and merging the tiles identified as duplicate tiles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Various aspects of the present disclosure relate to methods, apparatus, and systems for processing spatial data. Background Art

[0002] Spatial data, such as satellite image data, is increasingly used for various purposes and applications, such as vehicle navigation, nature exploration, and weather forecasting.

[0003] Spatial data, such as geographic maps, can be processed, rendered, and visualized on a web or internet browser using a method called tiling. Tiled geographic maps can be displayed by seamlessly joining multiple separately requested images. These separately requested images can also be called raster tiles.

[0004] Current methods for processing spatial data may not be sufficient to cope with exponentially growing data sets, which poses challenges in terms of storage, bandwidth, and maintenance. In addition, the processing of spatial data can cause a large amount of duplication in storage. Therefore, there is a need for improved processing and maintenance of spatial data. Summary of the invention

[0005] The technical solution seeks to provide a method, apparatus and / or system for processing spatial data (such as standard satellite imagery). In some embodiments, the processing includes slicing, updating storage and long-term maintenance to facilitate the creation of one or more self-hosted satellite map tile services, where users can request region-specific geographic map tiles, stitching and rendering them on demand. The output of the method, apparatus and / or system in the form of map tiles can be utilized by a navigation system installed in a vehicle (such as a car) for access by a user via one or more user interfaces.

[0006] According to one aspect of the present disclosure, a method for processing spatial data is provided, the method comprising the following steps: receiving a spatial data file from a spatial data source; generating a first tile set associated with the spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers; comparing the identifier of each tile of the plurality of tiles in the first tile set with the identifier of each tile in the second tile set to identify at least one repetition; and merging tiles identified as duplicate tiles.

[0007] In some implementations, each generated identifier may include zoom level data and location data associated with the corresponding tile.

[0008] In some embodiments, the method comprises the steps of generating a repetition indicator and appending the repetition indicator to the identifier, the repetition indicator being associated with the number of repeated tiles in the first tile set and the second tile set.

[0009] In some embodiments, the step of generating the first set of tiles includes combining a plurality of generated tiles associated with a higher zoom level to generate tiles associated with a lower zoom level.

[0010] In some embodiments, the method further includes the step of packaging the merged tiles into a file format for storing a tile set.

[0011] In some implementations, the merged tiles are stored in a designated directory.

[0012] In some embodiments, the step of merging tiles includes overlaying at least one repeating tile on top of another repeating tile.

[0013] According to another aspect of the present disclosure, a device for processing spatial data is provided, the device comprising: an input module configured to receive a spatial data file and a second tile set; a slicing module configured to generate a first tile set associated with the spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers; an update module configured to compare an identifier of each tile of a plurality of tiles in the first tile set with an identifier of each tile in the second tile set to identify at least one repetition; and merging tiles identified as duplicate tiles.

[0014] In some implementations, each generated identifier includes zoom level data and location data associated with the corresponding tile.

[0015] In some embodiments, the update module is configured to generate a repeat indicator and append the repeat indicator to the zoom level data and the position data, the repeat indicator being associated with a number of repeated tiles in the first tile set and the second tile set.

[0016] In some implementations, the slicing module is configured to generate the first set of tiles by combining a plurality of generated tiles associated with a higher zoom level to generate tiles associated with a lower zoom level.

[0017] In some implementations, the update module is configured to package the merged tiles into a file format for storing a tile set.

[0018] In some implementations, the update module is configured to store the merged tiles in a specific directory.

[0019] In some implementations, the update module is configured to merge the repeated tiles by overlaying at least one repeated tile on top of another repeated tile.

[0020] According to another aspect of the present disclosure, there is a system for processing spatial data, the system comprising the apparatus as described and a storage module, wherein the merged tiles are sent to the storage module.

[0021] In some embodiments, the system further comprises a Web Map Tile Service (WMTS) service, wherein the storage module comprises an interface allowing access by the Web Map Tile Service (WMTS).

[0022] In some embodiments, the system also includes a vehicle navigation system configured to access the WMTS to render map tiles on demand.

[0023] According to another aspect of the present disclosure, a system for processing spatial data is provided, the system comprising: a master node arranged to communicate data or signals with a plurality of slave nodes, the master node comprising a task scheduler to assign at least one task to each of the plurality of slave nodes; a first slave node configured to generate a first tile set associated with a first spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers; a second slave node configured to generate a second tile set associated with a second spatial data file, the second tile set comprising a plurality of tiles and corresponding identifiers; a processing module configured to compare an identifier of each tile of a plurality of tiles within the first tile set with an identifier of each tile in the second tile set to identify at least one repetition; and a third slave node configured to merge tiles identified as duplicate tiles.

[0024] In some implementations, each generated identifier includes zoom level data and location data associated with the corresponding tile.

[0025] In some embodiments, the update module is configured to generate a repeat indicator and append the repeat indicator to the zoom level data and the position data, the repeat indicator being associated with a number of repeated tiles in the first tile set and the second tile set.

[0026] In some embodiments, the system further comprises a storage module arranged to communicate data or signals with at least one of the first slave node, the second slave node, and the third slave node.

[0027] In some embodiments, the system further includes a pre-processing module to obtain a batch of spatial data and segment the batch of spatial data into at least a first spatial data file and a second spatial data file.

[0028] In some embodiments, the storage module is configured to identify at least one spatial data file associated with a specific file format from a batch of spatial data, so that the spatial data file associated with the specific file format is configured to bypass the pre-processing module and be sent to the third slave node.

[0029] In another aspect of the disclosure, a non-transitory computer-readable storage medium including instructions is provided, which, when executed by one or more processors, causes the method for processing spatial data according to any one of the above method embodiments to be performed.

[0030] In another aspect of the disclosure, a data processing device is provided, which is configured to perform a method according to any one of the above method embodiments.

[0031] In another aspect of the disclosure, a computer executable code is provided, the computer executable code including instructions for processing spatial data according to any one of the method embodiments described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and accompanying drawings, in which:

[0033] - Figure 1 is a flow chart associated with a method for processing spatial data according to various embodiments;

[0034] - FIG. 2A to FIG. 2C The concepts of tiles and zoom levels associated with slicing of spatial data files in the context of the present disclosure are shown;

[0035] - Figure 3 shows a system architecture including a tile module, a storage module, and a web map tile service (WMTS) according to various embodiments;

[0036] - Figure 4 shows a flow chart associated with a method for splitting and merging tiles;

[0037] - Figure 5 A distributed system for processing spatial data according to various embodiments is shown; and

[0038] - Figure 6 A schematic diagram of a processor for processing spatial data according to some embodiments is shown. DETAILED DESCRIPTION

[0039] The following specific embodiments refer to the accompanying drawings, which illustrate the specific details and embodiments in which the present disclosure can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. Other embodiments may be utilized, and structural changes and logical changes may be made without departing from the scope of the present disclosure. Various embodiments are not necessarily mutually exclusive, as some embodiments may be combined with one or more other embodiments to form new embodiments.

[0040] Embodiments described in the context of one of the housing systems, devices or methods are similarly valid for the other systems, devices or methods. Similarly, embodiments described in the context of the system are similarly valid for the device or method, and vice versa.

[0041] Features described in the context of one embodiment may be correspondingly applicable to the same or similar features in other embodiments. Features described in the context of one embodiment may be correspondingly applicable to other embodiments, even if not explicitly described in these other embodiments. In addition, additions and / or combinations and / or substitutions as described for features in the context of one embodiment may be correspondingly applicable to the same or similar features in other embodiments.

[0042] In the context of various embodiments, the articles “a,” “an,” and “the” as used with respect to features or elements include reference to one or more of the features or elements.

[0043] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0044] As used herein, the term "data" may be understood to include information in any suitable analog or digital form, for example, provided as a file, a portion of a file, a collection of files, a signal or stream, a portion of a signal or stream, a collection of signals or streams, etc. However, the term data is not limited to the above examples and may take various forms and represent any information as understood in the art.

[0045] As used herein, the term "spatial data" refers to data in various formats associated with one or more geographic locations or regions. Non-limiting examples of spatial data include satellite images, georeferenced maps in two-dimensional or three-dimensional form. Such spatial data can be stored in various file formats with or without metadata. Non-limiting examples of file formats include Geotiff, .png, and .jpg file formats.

[0046] As used herein, the term "metadata" broadly refers to data in various data formats that provide information about content, excluding the content. For example, metadata associated with an image may include information such as the image source, the image capture device, the actual location / coordinates in space, and a representation of each pixel in the image. Metadata can be added to existing metadata. In one example, metadata can be represented as an identifier or key in the form of Z / X / Y, where Z represents the map tile zoom level and X and Y represent the X and Y coordinates of the image on the plane axis, respectively. After processing or analysis, further information can be tagged or attached to the metadata to indicate the status, attributes, and / or parameters of the metadata relative to other metadata.

[0047] As used herein, the term "update" broadly includes any process that reduces at least one of processing time and / or data duplication without including data integrity. Examples of update operations may include merge operations, overwrite operations, partial rewrite operations, exception capture, log tracking, version control data rollback capabilities, and one or more combinations of the above.

[0048] As used herein, the term "module" refers to an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above (such as in a system on a chip), or forming part of, or including it. The term module may include a memory (shared, dedicated, or group) that stores code executed by a processor. A single module or a combination of modules may be considered a device.

[0049] As used herein, the term "node" refers to any computing device with processing and communication capabilities. Non-limiting examples of nodes include computers, mobile smart phones, computer servers.

[0050] As used herein, the terms "associate," "associated," "associate," and "associating" indicate a defined relationship (or cross-reference) between two items. For example, a tile set may be associated with a spatial data file, indicating that the tile set may be derived, calculated, and / or processed using the spatial data file as a source or reference. A tile set may be a subset of a spatial data file, or may include additional information, such as metadata.

[0051] As used herein, "memory" may be understood as a non-temporary computer-readable medium in which data or information may be stored for retrieval. Therefore, references to "memory" included herein may be understood to refer to volatile or non-volatile memory, including random access memory ("RAM"), read-only memory ("ROM"), flash memory, solid-state storage device, tape, hard disk drive, optical disk drive, etc., or any combination thereof. In addition, it should be understood that registers, shift registers, processor registers, data buffers, etc. are also included in the term memory herein. It should be understood that a single component referred to as "memory" or "a memory" may be composed of more than one different type of memory, and therefore may refer to a collective component including one or more types of memory. It is readily understood that any single memory component may be divided into a plurality of commonly equivalent memory components, and vice versa. In addition, although the memory may be depicted as being separated from one or more other components (such as in the accompanying drawings), it should be understood that the memory may be integrated within another component (such as integrated on a common integrated chip).

[0052] According to one aspect of the present disclosure and with reference to Figure 1 There is a method 100 for processing spatial data, the method comprising the following steps: receiving a spatial data file from a spatial data source (step S102); generating a first tile set associated with the spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers (step S104); comparing the identifier of each tile of the plurality of tiles in the first tile set with the identifier of each tile in the second tile set to identify at least one repetition (step S106); and merging tiles identified as duplicate tiles (step S108).

[0053] The spatial data file can be a satellite image having a file format (such as Geotiff) and stored on a storage module, which is configured to be accessed through a WMTS service. In some embodiments, the storage module can be configured to provide high persistence, high availability, and high security for the storage of satellite image tilesets. For example, such a storage module can be hosted by a cloud service, and / or can be implemented as a distributed database shared between different nodes, such as a blockchain. In some embodiments, the storage module can be periodically arranged to receive spatial data via a job scheduler.

[0054] The storage module can be configured to support content delivery network (CDN) acceleration of the satellite image tile set. A CDN can include a group of geographically distributed servers that work together to provide relatively fast Internet content delivery.

[0055] In step S102, a spatial data file may be obtained from a storage module.

[0056] In step S104, the generation of the first tile set may include the generation of a raster image and a corresponding identifier. The corresponding identifier may be appended to pre-existing metadata accompanying the source spatial data file, or if the source spatial data file does not include any metadata, the corresponding identifier may be newly generated. Examples of pre-existing metadata may include at least one of the following information about the image source, the image capture device, the actual position in space, and the representation of each pixel in the image. The generated identifier may be expressed in the form Z / X / Y, where Z represents the zoom level and X and Y represent a series of X coordinates and Y coordinates of the image on the plane axis, respectively. For the purpose of clarity, the generated identifier may be referred to as a "key" from now on.

[0057] In step S106, the second tile set may be associated with another spatial data file, which may include previously processed slice images that have been stored in the storage module. Comparing the identifiers to identify one or more repetitions may include comparing the same Z / X / Y keys. For example, both tile sets (a first tile set and a second tile set generated from two spatial data files) may include keys indicating "Z1 / X1 / Y1". This may correspond to two satellite images associated with the same location or approximately the same location (one image from the first tile set and one image from the second tile set).

[0058] In step S108, merging tiles associated with duplicate tiles based on the identifier may include an image processing step (such as an overlay operation) to combine or merge duplicate tiles. It is worth noting that the two duplicate tiles associated with the key "image-Z1 / X1 / Y1, (1, 1)" in the previous example will be combined. The newly combined tile can replace the original duplicate tile, and the original duplicate tile can be removed or deleted, thereby reducing memory resources.

[0059] In some implementations, the keys are represented and stored as integers, where the zoom level Z is an integer value ranging from 0 to 22, and X and Y represent specific segments of the divided map.

[0060] Using a world map as an example of spatial data to be processed, Z / X / Y=0 / 0 / 0 corresponds to zoom level 0, and since zoom level 0 does not include any divided tiles, there is only one spatial data file 0 / 0 / 0 associated with zoom level 0.

[0061] At zoom level 1, the world map is divided into Figure 2AThe four equal parts shown in , correspond to four keys Z / X / Y=l / 0 / 0, 1 / 0 / 1, 1 / 1 / 0, 1 / 1 / 1, each key corresponding to the map tile 202.

[0062] Key 1 / 0 / 0 may correspond to a map tile in the upper left corner. In this case, key 1 / 0 / 0 corresponds to a geographic interval with longitude values ​​from -180 to 0 (half of -180 to 180) and latitude values ​​from 0 to 90 (half of -90 to 90). Key 1 / 0 / 1 may correspond to a map tile in the upper right corner with a latitude range the same as 1 / 0 / 0 (0 to 90) and a longitude range of 0 to 180 (half of -180 to 180). Key 1 / 1 / 0 may correspond to a map tile in the lower left corner with a longitude range the same as 1 / 0 / 0 (-180 to 0) and a latitude range of -90 to 0. Key 1 / 1 / 1 may correspond to a map tile in the lower right corner with a longitude range of 0 to 180 and a latitude range of -90 to 0.

[0063] Each of the four tile images at Z=1 can be divided into four equal parts as before, so that there are a total of 16 keys at Z=2, each key corresponding to a map tile 204, such as Figure 2A as shown in .

[0064] In some embodiments, the comparison step in step S106 may include the step of generating a repeat indicator in the first tile set and the second tile set and appending the repeat indicator to the identifier (step S112). The repeat indicator may be in the form of a label and may be appended to the generated key. For example, in an example where both the first tile set and the second tile set contain tiles associated with the key Z1 / X1 / Y1, the repeat indicator (1,1) may be appended to the entry to form a modified identifier "Image-Z1 / X1 / Y1, (1,1)", with the label (1,1) associated with the two repeat entries. Another tile from the first tile set or the second tile set may include metadata indicating "Z3 / X3 / Y3", where the generated label (1) is appended to the identifier, i.e., "Image-Z3 / X3 / Y3, (1)". The label (1,1) means that there are two images with the same metadata in the generated tile set, and the label (1) means that there is only one image with the metadata in the tile set. In general, each time an additional repeating tile is found, an additional (1) is added to the label, ie, (1,1,1) is added for three repeating tiles, (1,1,1,1) is added for four repeating tiles, etc. It is contemplated that other forms of labeling may be considered.

[0065] FIG. 2A to FIG. 2CAn example of forming multiple tiles based on spatial data in the form of a world map is shown. In some embodiments, the multiple tiles may include square images, which may be in the form of a bitmap data format, including a map displayed in a grid-like arrangement. Tiles may be generated according to multiple zoom levels, each zoom level indicating how large or small the contents of the map appear in the map view. At zoom level 0, the entire world is located on a single tile.

[0066] like Figure 2A As shown in FIG. 2 , a single tile at zoom level 0 is split into four tiles 202 numbered 0 to 3 at zoom level 1 for global coverage. Each subsequent zoom level quadrilaterally divides the tile of the previous zoom level, thereby creating a tile consisting of 2 zoom ×2 zoom A grid of tiles. Figure 2B As shown in , the geographic extent remains the same from the bottom to the top of the tile pyramid. It is noteworthy that the number of tiles is fixed for each zoom level, i.e., zoom level 2 includes sixteen tiles 204, zoom level 3 includes sixty-four tiles 206, and the number of tiles at the next higher zoom level can be extrapolated or predicted by multiplying the number of tiles at the current zoom level by four, as Figure 2C as shown in .

[0067] Figure 3 A is a system architecture of a system 300 for processing spatial data, which includes a tile generator device (also referred to as a map tile maker) 302, a spatial data storage device 304, and a spatial data rendering service 306. Each of the above components can be independent, or various integrations between the components can be envisioned. For example, in some embodiments, the map tile maker device 302 can be combined with the spatial data storage device 304 to form an integrated device.

[0068] In some embodiments, the map tile maker device 302 can be a computing device, such as a desktop computer or laptop computer arranged to communicate data and / or signals with the spatial data storage device 304. The device 302 may include an input / output module 312, which is configured to receive spatial data and other input data from the spatial data storage device 304 and return processed spatial tiles to the spatial data storage device 304. The spatial data storage device 304 can be the storage module described. The spatial data can be in the form of a data file format Geotiff. The Geotiff file 308a comes from the spatial data storage module 304 for processing, and in some embodiments, the dependency 308b and the current (old) tile 308c can be received from the spatial data storage module 304. The output end of the device 302 sends the updated tile 308d to the spatial data storage module 304 after processing. In some embodiments, the dependency 308b may include inputs of software libraries (such as the Geospatial Data Abstraction Library (GDAL) and image processing libraries). The dependency file can be in a binary format compiled based on an operating system for Geotiff tile processing.

[0069] It is envisioned that a map tile maker device 302 can implement and execute method 100. The map tile maker device 302 may include a slicing module 314 and an updating module 316. The step of converting spatial data 308a into a plurality of tiles may be implemented by a slicing module 314, which also generates corresponding keys for each tile if necessary. The updating module 316 may be configured to compare each key of a plurality of tiles with an old tile 308c in the data storage module 304 to identify duplications and generate corresponding duplication indicators in the form of tags to be attached to each key. If duplications are found, the updating module 316 operates to merge tiles associated with duplicate identifiers to form updated tiles 308d. The updating module 316 may also implement the merging step S108. In some embodiments, the updating module 316 may include submodules such as, but not limited to, a data logger, an error handler, an automatic scaler, data rollback, and version control. The updated tile 308d may be used to replace one or more existing tiles in the spatial data storage module 304.

[0070] It is contemplated that the tile sets may be generated asynchronously and may not affect the operation of the spatial data storage module 304. Synchronization may occur between the map tile maker device 302 and the spatial data storage module 304 as needed and when desired.

[0071] In some embodiments, dependency 308b, such as the Geospatial Data Abstraction Library (GDAL), can be used to read and write raster and vector geospatial data formats. GDAL can include command-line utilities for data conversion and processing, such as, but not limited to: (a.) reading and writing of raster and vector geospatial formats; (b.) data format conversion; (c.) geospatial processing, such as: subsetting, image warping, reprojection, mosaic processing, slicing, digital elevation model (DEM) processing.

[0072] GDAL includes various library methods that can be used to facilitate the tiling of satellite imagery in Geotiff data format. An example of a library method is the gdal2tiles function, which is an extension to GDAL that can conform to standards such as OSGeo TM The Tile Map Service Specification) generates tilesets and metadata from Geotiff files. The generated tilesets can be used in various interactive web map platforms (such as but not limited to Leaflet TM , Google Maps TM and OpenLayers TM ) to display an image overlay on the

[0073] Binary files compiled based on the gdal2tiles source code can include parts of static libraries for data raster tile processing, and can also be cross-compiled for different operating systems via Platform as a Service (PaaS) or virtual machine technology.

[0074] The gdal2tiles function can support single-threaded tile processing and multi-threaded tile processing. Multi-threaded tile processing can be used in a distributed and / or parallel processing system, where multiple instances of generating tiles from a spatial data file using method 100 are running in parallel, for example.

[0075] gdal2tiles can be configured to receive input parameters. The input parameters can include resampling parameters; a conversion scheme or a specified file format, such as TMS or the XYZ Slippy Map standard; the zoom level to render; and the amount of processing to be used for tiling and / or accelerated calculations.

[0076] Figure 4 An example of a gdal2tiles function that generates multiple tiles is shown. Process 400 may include the following: Figure 4 The three steps shown in .

[0077] At step S402, the metadata of the Geotiff data may be parsed based on the selected input parameters, and the highest zoom level may be calculated, assigned or derived based on the input parameters related to the zoom level to be rendered. The highest zoom level is configured as part of the input of the TileJobInfo class of the gdal2tiles function.

[0078] At step S404: based on the input of the TileJobInfo class, the spatial data tile Geotiff data is split or cut to obtain a tile set with the highest zoom level. A tile set with a lower zoom level can be generated from a tile set with a higher zoom level that is generated faster. Since the tile set with the highest zoom level can only be obtained from the Geotiff file using GDAL, their speed may be affected by the size of the Geotiff file. In general, the lower the zoom level of the tile set obtained from the Geotiff, the larger the geographic area obtained by a single cut, and the slower the acquisition speed. For example, the time to generate a tile with a zoom level of 17 from a Geotiff will be much longer than the time to obtain 4 tiles with a zoom level of 18 from the Geotiff and merge them. Therefore, it is computationally more economical to generate tiles at a higher zoom level and then derive tiles at a lower zoom level by merging 4 or more tiles.

[0079] At step S406: the Create_overview_tiles function is used to generate a tileset of a low zoom level by merging tilesets of a high zoom level.

[0080] Figure 5 A distributed system 500 for processing spatial data according to various embodiments is shown. The distributed system 500 may have a multi-node system architecture for processing and generating large data sets using parallel, distributed algorithms on a server cluster. The distributed system 500 is suitable for processing large amounts of spatial data, such as satellite imagery, relatively quickly. An example of such a programming model is "MapReduce". TM A "system" is provided that facilitates processing by running various instances of method 100 in parallel, managing all communications and data transfers between the various parts of system 500, and providing redundancy and fault tolerance.

[0081] Distributed system 500 can include multiple nodes, and these multiple nodes are configured to perform various tasks, such as generating map tiles, comparing map tiles to detect one or more repetitions and combining repeated map tiles to realize data compression and promote data updating.At least one node can be configured to assign tasks to other nodes and monitor the processing of other nodes.As shown in the figure, this system can include master node 502 and multiple worker nodes 504 associated with the first task and multiple worker nodes 506 associated with the second task.Master node 502 is arranged to carry out data or signal communication with each worker node 504,506.Master node 502 includes task monitor---also referred to as job tracker.Job tracker is responsible for resource monitoring and job scheduling.Job tracker monitors the state of all worker nodes 504,506 and the job or task that each worker node 504,506 performs, and in the case of one or more worker nodes 504,506 failures, reassigns job or task to other worker nodes 504,506. The job tracker may also track the execution process, resource usage, and other information of the tasks, and send such data to the task scheduler 508 .

[0082] The task scheduler 508 may be a detachable module or a plug-in module. The task scheduler 508 may be implemented as a node separate from the master node 502. The task scheduler 508 may be configured according to the user's (multiple) requirements. The scheduler 508 is configured to select tasks that use node resources when the node resources become available.

[0083] Each worker node 504, 506 may be configured as a slave node arranged to communicate data or signals with the master node 502 to implement one or more instances of processing spatial data. Figure 5 In the embodiment shown in , a first group of n worker nodes 504 are configured to perform a first task, i.e., steps associated with converting a spatial data file into corresponding tiles and metadata (also known as a mapping task), and a second group of n worker nodes 506 are configured to perform a second task, i.e., steps associated with merging duplicate tiles (also known as a simplification task).

[0084] The task scheduler 508 is arranged to communicate data / signals with the master node 502 to receive node and task related information and schedule tasks to slave nodes 504, 506. The task scheduler 508 can be a dynamic task scheduler. Each task can be a mapping task or a simplification task using a mapping function (such as a gdal2tile function). In some embodiments, the worker node can be responsible for executing these two tasks. The worker nodes 504, 506 periodically report any computer or memory resource usage and the process of the tasks being performed on the node to the master node 502 via one or more periodic signals also referred to as heartbeat signals, and receive commands from the master node 502 to perform corresponding actions, such as executing new tasks, stopping current tasks, terminating current tasks, etc.

[0085] In operation, the master node 502 assigns one or more "slots" to each worker node 504, 506 via the task scheduler 508. A "slot" represents a computing resource, such as a CPU resource in the form of computing power, a memory resource in the form of RAM, ROM, etc. A task can be assigned a slot before it is executed, and the master node 502 is operable to allocate available items for use by the worker nodes 504, 506. In some embodiments, two types of slots (map slots and reduce slots) are used for map tasks and reduce tasks, respectively. The task scheduler 508 limits the concurrency of tasks by the number of slots. The number of slots can be a customizable parameter or can be predefined by the user.

[0086] Next, the system 500 is described in the context of its operation with reference to four phases: a split phase, a map phase, a shuffle phase, and a reduce phase. The system 500 is particularly suitable for processing relatively large numbers of spatial data files in a distributed and parallel architecture. The spatial data files may be in a variety of data formats.

[0087] Segmentation phase - In the segmentation phase, spatial data files, which may be in the form of satellite images in different data formats, are received from a source storage device 510 in the form of a batch input file 512. The batch spatial data file is segmented to obtain corresponding individual spatial data input files. The size of each individual spatial data file may not exceed a specific block size. The specific block size may be a configurable or predefined parameter.

[0088] In some embodiments, the spatial data of the tile determined to be attached with metadata (such as a Geotiff tile with metadata) is treated as an atomic file, that is, no segmentation operation is performed on the input individual Geotiff tile. Such an atomic file is directly sent to the worker node 506 assigned to the reduction task. Such a configuration can reduce the recalculation and segmentation time of the image, geographic and metadata information of the Geotiff file.

[0089] Each generated segmentation item is assigned to a map worker 504 to generate multiple tiles.

[0090] Mapping phase - using the assigned segmented Geotiff file as input, each worker node 504 is assigned to generate multiple tiles based on method 400. The worker node 504 applies method 400 to generate the file and writes the generated output tiles to a temporary storage device, which can be a cache. The master node 502 can be configured to determine that only one copy of the input spatial file is processed.

[0091] When the worker node 504 receives the segmented Geotiff data and performs a mapping task to generate multiple tiles, the worker node 504 can periodically send a heartbeat signal to the master node 502 to indicate its progress. Once the worker node 504 completes the mapping task, it can send a heartbeat signal indicating that it is in a state ready to execute a new task, and the master node 502 can assign a new mapping task to the worker node 504.

[0092] In some embodiments, the mapping task can be assigned based on a scheduling logic (such as Data-Local), where the mapping task can be assigned to a worker node 504 containing the segmented data blocks required for the mapping task, and the segmented spatial data file (in the form of a mapping package) can be copied to the assigned worker node 504. The mapping package can be a custom mapping function that is used to process the data input to the mapping task by calling gdal2tiles in this case by calling a first tile set (multiple) for parallel processing of Geotiff to generate an image and corresponding metadata. In summary, the mapping stage is mainly responsible for parallel slicing of batches of Geotiff satellite images to generate raster tile sets, which are regarded as intermediate tile files.

[0093] Shuffle phase - In the shuffle phase, the processor can be used to obtain intermediate tile files from the worker node 504. Then, operations such as partitioning, sorting, and combining are performed. Such operations are based on parsing metadata to generate relevant keys in the form of Z / X / Y, where Z represents the zoom level, and X and Y represent the X coordinate and Y coordinate of the image on the plane axis, respectively. The partitioning operation can be performed according to different sharding logics (e.g., the number of keys Z / X / Y hashcode% of the simplification worker), and the intermediate tiles generated in the mapping phase are stored in different partitions based on their keys. The sorting operation sorts the intermediate files output by the keys of the mapping phase. The combination operation merges the duplicate key values ​​of the intermediate results generated by the mapping phase. The intermediate results refer to all images in the tile set generated from the mapping phase, where the key is Z / X / Y. It is worth noting that each image has a unique Z / X / Y key, and different satellite image tiles can generate the same Z / X / Y image in the tile set. Therefore, the combination operation will record the merge operation of duplicate key values ​​from the tile set.

[0094] In some embodiments, the hash function may include any (multiple) functions that can be used to associate spatial map data of any size with a fixed size value. The final fixed size mapping result is represented in the form of a hash code. Different hash codes can be generated by processing different data through a hash function. For the same data, the hash code obtained by the same hash method each time will be the same to ensure consistency. The hash code can be represented in the form of a string data type. In some embodiments, the hash code can be divided by an integer and the remainder is found (for example, using a mathematical operation modulo). The integer can be the number of worker nodes 504, 506. By using the above-mentioned string modulo method, spatial data tiles can be evenly distributed to different workers depending on Z / X / Y.

[0095] For example, in an embodiment, there are two reducers 506 and four spatial data files Z1 / X1 / Y1, Z2 / X2 / Y2, Z3 / X3 / Y3, Z4 / X4 / Y4. Applying the same hash function or method,

[0096] Z1 / X1 / Y1 generates hashcode1 through hash method, and hashcode1 string modulo 2=0.

[0097] Z2 / X2 / Y2 generates hashcode2 through the hash method, and the hashcode2 string modulo 2=1.

[0098] Z3 / X3 / Y3 generates hashcode3 through hash method, and hashcode3 string modulo 2=1.

[0099] Z4 / X4 / Y4 generates hashcode4 through the hash method, and the hashcode4 string modulo 2=0.

[0100] Then, Z1 / X1 / Y1 and Z4 / X4 / Y4 will be processed by the first reduce worker 506 , and Z2 / X2 / Y2 and Z3 / X3 / Y3 will be reduced by the second reduce worker 506 .

[0101] If the tilesets generated by both mapper 1 and mapper n include the Z1 / X1 / Y1 image, the combine operation will generate a duplicate indicator (1,1) and record the duplicate image as image-Z1 / X1 / Y1,(1,1). As another example, if only the tileset generated from mapper n includes the Z3 / X3 / Y3 image, the combine operation will generate a duplicate indicator (1) - indicating that there is no duplication, and record image-Z3 / X3 / Y3,(1). In the above example, (1,1) means that there are two images with that key in the generated tileset, and (1) means that there is only one image with that key in the tileset.

[0102] It is contemplated that the combined operation may achieve at least one of data aggregation, data race avoidance, data consistency, and idempotence of subsequent reduce phases to perform the task.

[0103] In some embodiments, the images in the tile set generated by the worker node 504 can be transmitted over a remote network to or easily read from the local resources of the worker node 506 assigned to the reduce task according to the key Z / X / Y partition result. For example, the image -Z2 / X2 / Y2,(1) generated by the map worker 504 of node 1 is transmitted to the reduce worker 506 of node n. In some embodiments, the key repeat image can be transmitted to a reduce worker 506 or read locally by the reduce worker, for example, the image -Z1 / X1Y1,(1,1) is directly read from the local by the reduce worker 1 of node 1 as input data of the reduce task.

[0104] Simplification phase - The simplification phase can use computing and storage resources through unified planning and scheduling of simplification tasks, and achieve the purpose of incrementally updating the generated tile set to the existing tile set in the storage device. The tasks of the simplification phase are also performed by a custom simplification package. In the simplification phase, it is first determined whether the image corresponding to the key (Z / X / Y) is located in the storage device, and if it exists, the image is downloaded by referencing the key in the storage device to the node and calling the merge method of the image processing library to merge the image with the image of the same key of the simplification task, and finally the output image is uploaded to the storage device. If the image of the key (Z / X / Y) does not exist in the storage device, the images with the same key are first merged locally in the node (if there is only one image for the key in the node, no merging is required), and then the resulting image is uploaded to the storage device. When all simplification tasks are completed, the entire system completes the distributed processing of batch Geotiff satellite image data and the incremental update of the existing satellite tile set.

[0105] Figure 6 A server computer system 600 is shown according to various embodiments. The server computer system 600 includes a communication interface 602 (e.g., configured to receive input data from the first sensor 104). The server computer 600 also includes a processing unit 604 and a memory 606. The processing unit 604 can be used by the memory 606 to store, for example, data to be processed, such as data associated with the input data and the results output from the modules 202, 204, 206. The server computer is configured to execute Figure 1 and / or Figure 4 It should be noted that the server computer system 600 may be a distributed system including a plurality of computers. The memory 606 may include a non-transitory computer-readable medium.

[0106] In various embodiments utilizing an image processing step, an image processing library, i.e., a free and open source cross-platform software suite for displaying, creating, converting, modifying, and editing raster images, can be used. Specifically, the conversion program in the CLI can be used to convert between image formats and perform various image processing operations, such as resizing, blurring, cropping, despeckle, dithering, drawing, flipping, splicing, resampling, and more. In various described embodiments, the image processing library can be utilized to overlay and merge multiple images into a single image.

[0107] In the described embodiment, it is envisioned that the generated tileset or updated tileset can be obtained by selecting to output to a specified directory or to be packaged into an mbtile file format. MBTiles is a file format for storing raster or vector tilesets in a SQLite database.

[0108] It is contemplated that the output of the described methods, systems and / or apparatus may be deployed in a navigation system for access by a vehicle (such as a car). Specifically, the generated updated map tiles may be evaluated by a navigation system installed in a car with a user interface (such as a touch screen) to render a map including the processed spatial data.

[0109] The methods described herein may be performed, and the various processing or computing units and devices and computing entities described herein may be implemented by one or more circuits. In an embodiment, a "circuit" may be understood as any kind of logic implementation entity, which may be hardware, software, firmware, or any combination thereof. Thus, in an embodiment, a "circuit" may be a hardwired logic circuit or a programmable logic circuit, such as a programmable processor, e.g., a microprocessor. A "circuit" may also be software implemented or executed by a processor, e.g., any kind of computer program, e.g., a computer program using virtual machine code. Any other kind of implementation of the corresponding functions described herein may also be understood as a "circuit" according to an alternative embodiment.

[0110] Although the present disclosure has been specifically shown and described with reference to certain embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims. Therefore, the scope of the present disclosure is indicated by the appended claims, and therefore it is intended to cover all changes that fall within the meaning and range of equivalents of the claims.

Claims

1. A method for processing spatial data, comprising the following steps: receiving a spatial data file from a spatial data source; generating a first tile set associated with the spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers; comparing an identifier of each tile in a plurality of tiles in the first tile set with an identifier of each tile in a second tile set to identify at least one duplication; as well as Merge tiles that are identified as duplicate tiles.

2. The method according to claim 1, wherein: Each generated identifier includes zoom level data and position data associated with the corresponding tile. 3 . The method of claim 2 , further comprising the step of generating a repetition indicator and appending the repetition indicator to the identifier, the repetition indicator being associated with the number of repeated tiles in the first tile set and the second tile set.

4. A method according to any one of the preceding claims, wherein: The step of generating the first set of tiles comprises combining a plurality of generated tiles associated with a higher zoom level to generate tiles associated with a lower zoom level.

5. The method according to any one of the preceding claims, further comprising the step of packaging the merged tiles into a file format for storing a tile set.

6. A method according to any one of the preceding claims, wherein: The merged tiles are stored in the specified directory.

7. A method according to any one of the preceding claims, wherein: The step of merging the tiles includes overlaying at least one repeated tile on another repeated tile.

8. A device for processing spatial data, comprising: An input module configured to receive a spatial data file and a second tile set; A slicing module configured to generate a first tile set associated with the spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers; Update module, configured as: comparing the identifier of each tile of the plurality of tiles in the first tile set with an identifier of each tile in a second tile set to identify at least one repetition; as well as Merge tiles that are identified as duplicate tiles.

9. The device according to claim 8, wherein: Each generated identifier comprises zoom level data and position data associated with the corresponding tile, wherein the update module is optionally configured to generate a repetition indicator and append the repetition indicator to the zoom level data and the position data, the repetition indicator being associated with the number of repeated tiles in the first tile set and the second tile set.

10. The device according to any one of claims 8 or 9, wherein: The slicing module is configured to generate the first set of tiles by combining a plurality of generated tiles associated with a higher zoom level to generate tiles associated with a lower zoom level.

11. The device according to any one of claims 8 to 10, wherein: The update module is configured to package the merged tiles into a file format for storing a tile set, or the update module is configured to store the merged tiles in a specific directory, and / or wherein the update module is optionally configured to merge the repeated tiles by overlaying at least one repeated tile on top of another repeated tile.

12. A system for processing spatial data, comprising the device according to any one of claims 8 to 11, and a storage module, wherein: The merged tiles are sent to the storage module.

13. The system of claim 12, further comprising a Web Map Tile Service (WMTS) service, wherein: The storage module includes an interface allowing access by a Web Map Tile Service (WMTS), wherein the system optionally includes a vehicle navigation system configured to access the WMTS to render map tiles on demand.

14. A system for processing spatial data, comprising: a master node arranged to communicate data or signals with a plurality of slave nodes, the master node comprising a task scheduler to assign at least one task to each of the plurality of slave nodes; A first slave node configured to generate a first tile set associated with a first spatial data file, the first tile set comprising a plurality of tiles and corresponding identifiers; a second slave node configured to generate a second tile set associated with a second spatial data file, the second tile set comprising a plurality of tiles and corresponding identifiers; a processing module configured to compare the identifier of each tile of the plurality of tiles in the first tile set with the identifier of each tile in the second tile set to identify at least one duplication, and The third slave node is configured to merge tiles identified as duplicate tiles.

15. The system of claim 14, wherein: Each generated identifier comprises zoom level data and position data associated with the corresponding tile, and wherein the update module is optionally configured to generate a repetition indicator and append the repetition indicator to the zoom level data and the position data, the repetition indicator being associated with the number of repeated tiles in the first tile set and the second tile set.

16. The system according to any one of claims 14 or 15, further comprising a storage module arranged to communicate data or signals with at least one of the first slave node, the second slave node and the third slave node, and the system optionally comprises a pre-processing module to obtain a batch of spatial data and to divide the batch of spatial data into at least the first spatial data file and the second spatial data file.

17. The system of claim 16, wherein: The system includes a pre-processing module, and the storage module is configured to identify at least one spatial data file associated with a specific file format from the batch of spatial data, so that the spatial data file associated with the specific file format is configured to bypass the pre-processing module and be sent to the third slave node.

18. A non-transitory computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the method for processing spatial data according to any one of claims 1 to 7 to be performed.

19. A data processing device configured to execute the method according to any one of claims 1 to 7.

20. Computer executable code comprising instructions for processing spatial data according to any one of claims 1 to 7.