Method and system for creating and using partitioned geodata marts

Abstract

The invention relates to a method and system for creating partitioned geodata marts. The method entails: receiving, as input, data that include geodata, a list of numerical values from a minimum to a maximum value of the scale levels of the data marts to be created, and a dataset containing arbitrary attributes for the creation of a data mart; combining the received geodata with the dataset; creating, on the basis of the outcome of the preceding step, tiles and tile identifiers for all scale levels of the data marts to be created, as follows: for the current scale level in a cycle, using the tile identifiers to obtain all of the data for a given tile that are to be found in the combined dataset, then converting the tiles into a format for processing by the data mart and saving said tiles in the memory of a data storage device; using the resulting tiles in the operation of a data warehouse in order to create partitioned data marts containing geodata by: selecting the desired scale level, determining all of the tile identifiers at the current scale level, requesting tiles according to the determined identifiers, and creating and visualizing the corresponding data marts. The invention makes it possible to increase the efficiency with which tiles are identified.

Description

[0001] METHOD AND SYSTEM FOR FORMING AND USING PARTITIONED GEODATA SHOWCASES

[0002] AREA OF TECHNOLOGY

[0003] [1] This technical solution generally relates to methods of processing data and preparing data structures for viewing and visualization, namely to a method of forming partitioned data marts containing geodata and their use in the process of operating a data warehouse.

[0004] LEVEL OF TECHNOLOGY

[0005] [2] Currently, a method of processing data in data warehouses is known in which three standard data layers are logically distinguished:

[0006] • Initial (stage);

[0007] • Normalized (detailed);

[0008] • Adapted for viewing (showcases).

[0009] [3] As a rule, separate technical solutions are used to process each of the layers. Storefronts are often partitioned (sharded, for example, on different database instances) by some features, such as period, client id, and in general, by any other attribute.

[0010] [4] In any data mart, when it comes to big data, partitioning is usually used. In particular, in geo marts, data is partitioned by assigning data to one or another geographic object.

[0011] 15] The technical solution described in is known from the prior art

[0012] RU 2681361 "USER INTERFACE FORMATION SYSTEM FOR INPUT, DISPLAY AND MODIFICATION OF VECTOR SPATIAL DATA", 18.04.2018, Federal State Treasury Military Educational Institution of Higher Education, the Academy of the Federal Guard Service of the Russian Federation (RU): "the user interface formation system for input, display and modification of vector spatial data additionally comprises a control unit for processing a set of initial vector spatial data, a unit for generating program code for the functional parts of the user interface, a unit for storing metadata about the initial vector spatial data, a unit for storing intermediate data, a unit for storing metadata about the output vector spatial data."

[0013] [6] The technical solution described in US11036752B2 “Optimizing incremental loading of warehouse data”, 08.08.2016, Oracle International Corp. (US) is known from the prior art: it presents an optimization of incremental loading of data into a database by identifying affected partitions and disabling only these partitions for the duration of the data loading; loading and storing data is also described.”

[0014] [7] The disadvantages of known solutions are the fact that when processing data and partitioning it, each partition has a format that can only be handled by a specific database management system in which the partitioning process was carried out, which imposes significant restrictions on their use. DISCLOSURE OF INVENTION

[0015] [8] This technical solution is aimed at eliminating the shortcomings inherent in existing solutions known from the prior art.

[0016] [9] The technical problem solved in this technical solution is that when processing data and partitioning it, each partition has a format that can only be worked with by the specific database management system in which the partitioning process was carried out, which imposes significant restrictions on their use.

[0017]

[0010] The main technical result that emerges from solving the above-mentioned problem is the expansion of the arsenal of technical means by providing the ability to create and use partitioned data in the form of static and ready-to-transfer files.

[0018]

[0011] Additional technical results that appear when solving the above-mentioned problem are acceleration of data processing, acceleration of integration of data storage modules, and acceleration of data caching at various levels of their processing when working with partitioned data.

[0019]

[0012] The specified technical results are achieved by implementing a method for forming partitioned data marts containing geodata and using them in the process of operating a data warehouse, implemented using a processor and a data storage device, including the following steps:

[0020] • receive input data including: geodata, a list of numerical values ​​from the minimum to the maximum value of the scale level of the generated data marts, and a dataset with arbitrary attributes for generating a data mart;

[0021] • combine the obtained geodata and dataset; • based on the result of the combination carried out in the previous step, form tiles and their identifiers for all scale levels of the formed data marts as follows: o for the current scale level in a cycle by tile identifiers, all available data for this tile in the combined dataset are obtained; o transform the tiles into a format for processing by the data mart and save them in the memory of the data storage device.

[0022] • use the tiles obtained during the operation of the data warehouse to form partitioned data marts containing geodata in the following way: o select the required zoom level; o automatically determine all tile identifiers at the current zoom level; o request tiles according to the defined identifiers, create and visualize the corresponding data marts.

[0023]

[0013] In one of the particular examples of implementing the method, when forming tiles and their identifiers, additional calculations are performed on the data, allowing the attributes to be changed.

[0024]

[0014] In another particular example of implementing the method, the selection of the zoom level is carried out through preset values.

[0025]

[0015] In another particular example of implementing the method, the selection of the zoom level is carried out dynamically.

[0026]

[0016] In addition, the claimed technical result is achieved by a system for forming partitioned data marts containing geodata and using them in the process of operating a data warehouse, comprising: o at least one data processing device; o at least one data storage device; o at least one program, where one or more programs are stored on one or more data storage devices and executed on one or more data processing devices, wherein one or more programs ensure the execution of the following steps:

[0027] • receive input data including: geodata, a list of numerical values ​​from the minimum to the maximum value of the scale level of the generated data marts, and a dataset with arbitrary attributes for generating a data mart;

[0028] • combine the obtained geodata and dataset;

[0029] • based on the result of the merger carried out in the previous step, tiles and their identifiers are formed for all scale levels of the formed data marts as follows: o for the current scale level, in a cycle by tile identifiers, all available data for this tile in the merged dataset are obtained; o the tiles are converted into a format for processing by the data mart and saved in the memory of the data storage device.

[0030] • use the tiles obtained during the operation of the data warehouse to form partitioned data marts containing geodata in the following way: o select the required zoom level; o automatically determine all tile identifiers at the current zoom level; o request tiles according to the defined identifiers, create and visualize the corresponding data marts.

[0031]

[0017] In one of the particular examples of the method implementation, when forming tiles and their identifiers, additional calculations are performed on the data, allowing the attributes to be changed.

[0018] In another particular example of the method implementation, the selection of the zoom level is carried out through preset values.

[0032]

[0019] In another particular example of implementing the method, the selection of the zoom level is carried out dynamically.

[0033] BRIEF DESCRIPTION OF DRAWINGS

[0034]

[0020] The features and advantages of the present technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[0035]

[0021] Fig. 1 illustrates a block diagram of the implementation of the claimed method.

[0036]

[0022] Fig. 2 illustrates an example of the operation of the described technical solution at a scale level from 0 to 1.

[0037]

[0023] Fig. 3 illustrates an example of the operation of the described technical solution at scale level 2.

[0038]

[0024] Fig. 4 illustrates an example of the operation of the described technical solution at scale level 13.

[0039]

[0025] Fig. 5 illustrates a system for implementing the claimed method. IMPLEMENTATION OF THE INVENTION

[0040]

[0026] Below, the terms and concepts necessary for the implementation of this technical solution will be described.

[0041]

[0027] Partitioning is the division of stored database objects (such as tables, indexes, materialized views) into separate parts with separate physical storage parameters. It is used to improve manageability, performance, and availability for large databases. Possible data partitioning criteria used in partitioning

[0042] — by predefined value ranges, by value lists, using hash function values; in some cases, other options are used. Composite (composite) partitioning criteria are understood as sequentially applied criteria of different types. Unlike segmentation, where each segment is managed by a separate DBMS instance, and means of coordination between them are used (which allows distributing the database across several computing nodes), with partitioning, access to all sections is carried out from a single DBMS instance (or symmetrically from any cluster DBMS instance, such as Oracle RAC).

[0043]

[0028] Tile, tiled or sign-based graphics (from English, tile - tile)

[0044] — a method of creating large images (usually levels in computer games) from small fragments of the same size. Tiles are small images of the same size that serve as fragments of a large picture. The number of tiles per "world" can reach several hundred. The cell matrix stores only the tile numbers, which saves memory when constructing huge two-dimensional spaces.

[0045]

[0029] A data set or dataset is a collection of data. In the case of tabular data, a data set corresponds to one or more database tables, where each column of the table corresponds to a different variable and each row corresponds to a record in the data set. Data sets store values ​​for each variable, such as the height and weight of an object, for each member of the data set. Data sets can also consist of a collection of documents or files.

[0046]

[0030] The scale level is a list of numerical values ​​from the minimum to the maximum value agreed upon in advance between the end user and the geoshowcase formation system. Viewing of the data of the showcases is possible only at one of the scale levels defined in this list. The method is suitable for various geodetic projections, but the projection of geoobjects onto the plane used in a specific implementation of the method must also be agreed upon in advance.

[0047]

[0031] The claimed technical solution may be implemented, for example, by a system, a machine-readable medium, a server, etc. In this technical solution, a system means, among other things, a computer system, a computer (electronic computer), a CNC (numerical control), a PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[0048]

[0032] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs).

[0049]

[0033] The command processing unit reads and executes machine instructions (programs) from one or more data storage devices, such as random access memory (RAM) and / or read-only memory (ROM). ROM may include, but is not limited to, hard disk drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.

[0034] A program is a sequence of instructions intended for execution by a computer control unit or a command processing unit.

[0050]

[0035] The term "instructions" as used in this application may generally refer to software instructions or software commands that are written in a given programming language to perform a specific function, such as, for example, receiving and processing data, forming a user profile, receiving and transmitting signals, analyzing received data, identifying a user, etc. The instructions may be implemented in a variety of ways, including, for example, object-oriented methods. For example, the instructions may be implemented using the C++ programming language, Java, Python, various libraries (e.g., Microsoft Foundation Classes), etc. The instructions that perform the processes described in this solution may be transmitted via both wired and wireless data transmission channels, such as Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.

[0051]

[0036] The presented method for the formation of partitioned data marts containing geodata and their use in the process of operating a data warehouse (Fig. 1 shows a diagram of the method) solves the problems of ensuring the ability to create and use partitioned data in the form of static and ready-to-transfer files, as well as accelerating data processing, accelerating the integration of data warehouse modules and accelerating data caching at various levels of their processing when working with partitioned data due to the sequential execution of the following steps:

[0052] • receive input data including: geodata, a list of numerical values ​​from the minimum to the maximum value of the scale level of the generated data marts and a dataset with arbitrary attributes to generate a data mart; • combine the received geodata and dataset;

[0053] • based on the result of the merger carried out in the previous step, tiles and their identifiers are formed for all scale levels of the formed data marts as follows: o for the current scale level, in a cycle by tile identifiers, all available data for this tile in the merged dataset are obtained; o the tiles are converted into a format for processing by the data mart and saved in the memory of the data storage device.

[0054] • use the tiles obtained during the operation of the data warehouse to form partitioned data marts containing geodata in the following way: o select the required zoom level; o automatically determine all tile identifiers at the current zoom level; o request tiles according to the defined identifiers, create and visualize the corresponding data marts.

[0055]

[0037] In the example of the method implementation below (functions in JS language with indication of the toolkit and examples of the initial and resulting data) GeoJSON with the wgs84 coordinate system was used as geodata. But it is possible to use any format (kml, csv, avro, parquet, ...) with any coordinate system. function.js - combining geodata and dataset, forming a geoshowcase: And the node:fs library allows interaction with the file system const fs = require('fs');

[0056] / / Library for slicing GeoJSON data into vector tiles var geojsonVt = require('geojson-vt'); / / Library for converting text data to binary (Protocol Buffers) var vtpbf = require('vt-pbf);

[0057] / / Library for defining a bounding box for geo objects var bbox = require('@turf / bbox');

[0058] And Leaflet library function for WGS84 coordinate transformation

[0059] / / in tile coordinates of a cylindrical coordinate system function project(lat, Ing, zoom) { function tiled(num) { return Math.floor(num / 256);

[0060] } var R = 6378137; var sphericalScale = 0.5 / (Math.PI * R); var d = Math.PI / 180, max = 1 - IE-15, sin = Math.max(Math.min(Math.sin(lat * d), max), -max), scale = 256 * Math.pow(2, zoom); var point = { x: R * Ing * d, y: R * Math.log((l + sin) / (1 - sin)) / 2

[0061] }; point.x = tiled(scale * (sphericalScale * point.x + 0.5)); point.y = tiled(scale * (-sphericalScale * point.y + 0.5)); return point;

[0062] / / Load JSON with sample data (without geometry) let current_dimension = JSON.parse(fs.readFileSync('data_dimension.json', 'utf8'));

[0063] / / Load Geo JSON (geometry only) let current_geojson = JSON.parse(fs.readFileSync('data_geo.geojson', 'utf8'));

[0064] / / Combine geometry and sample data current_geojson.features.forEach(f => {

[0065] / / Example for one indicator, but there can be as many of them as you like

[0066] Гргорекйе8['Customer Share'] = current_dimensionff.properties.name];

[0067] });

[0068] / / Load the result into the tile generation library let tileindex = geojsonVt(current_geojson);

[0069] / / Calculate the Border Box, the smallest rectangle,

[0070] / / which contains all objects let final bbox = bbox.default(current_geojson);

[0071] / / Create a directory for the resulting files fs.mkdirSync('output', {recursive: true});

[0072] / / Tile cutting is performed for each required zoom level

[0073] / / In this example from zero to six for (let zoom - 0; zoom <= 6; zoom++) { / / Find the minimum and maximum coordinates of the tile

[0074] / / along the X and Y axes containing the data,

[0075] / / for the current zoom level var min = project(final_bbox[3], final_bbox[0], zoom); var max = project(fmal_bbox[l], final_bbox[2], zoom);

[0076] / / Loop along the X axis for (var x = min.x; x <= max.x; x++) {

[0077] / / Loop along the Y axis for (var y = min.y; y <= max. y; y++) {

[0078] _ / / Form a vector with coordinates z, x, y let tile = tileindex.getTile(zoom, x, y);

[0079] _ / / Convert the tile to Protocol Buffers format let buff = vtpbf.fromGeojsonVt({'geojsonLayer': tile});

[0080] _ / / Write the tile to disk fs.writeFileSync('output / tile_$ {zoom}_$ {x}_$ {у} .pbf , buff);

[0081] } console. log(ToTOBo'); data dimension.json - data example (dataset): { "Adygea": 0.5201556202979407,

[0082] "Ingushetia": 0.20908676377182478,

[0083] "Stavropol Krai": 0.9055996348532036,

[0084] "Dagestan": 0.5168264313197202,

[0085] "Kabardino-Balkaria": 0.9424843279944837,

[0086] "Karachay-Cherkessia": 0.6759521775003527,

[0087] "Rostov region": 0.039308707400040444,

[0088] "Volgograd region": 0.928614174312159,

[0089] "Astrakhan region": 0.31795951295534564,

[0090] "Chechnya": 0.7758215042995391,

[0091] "North Ossetia - Alania": 0.09518987669413037,

[0092] "Krasnodar region": 0.7973789064618064,

[0093] "Republic of Kalmykia": 0.12141737873392788

[0094] } data geo.geojson - geometry example:

[0095] "type": "FeatureCollection",

[0096] "features": [

[0097] "type": "Feature",

[0098] "bbox": [

[0099] 38.68402, 43.76015,

[0100] 40.77643,

[0101] 45.21698

[0102] ],

[0103] "properties": {

[0104] "name": "Adygea"

[0105] },

[0106] "geometry": {

[0107] "type": "MultiPolygon",

[0108] "coordinates": [

[0109] [

[0110] [

[0111] [

[0112] 40.29454,

[0113] 43.7706

[0114] ],

[0115] [

[0116] 40.2617,

[0117] 43.76015

[0118] ],

[0119] And so on with further geo-coordinates. Run the function and the result: $ node function.] s

[0120] Ready:

[0121] $ find . / output -type f | sort

[0122] . / output / tile_0_0_0.pbf

[0123] . / output / tile_ 1 ​​> 1 _0.pbf

[0124] . / output / tile_2_2_l .pbf

[0125] . / output / tile_3_4_2.pbf

[0126] . / output / tile_3_5_2.pbf

[0127] . / output / tile_4_ 10_5.pbf

[0128] And so on with further partitioned data (tiles) in the form of separate files.

[0129]

[0038] The described technical solution proposes to store and use partitioned data in the form of static files ready for transfer to the user.

[0130] Benefits of using files:

[0131] 1. File handling is one of the basic tools of most OS, and is supported by the kernel. No additional tools need to be developed

[0132] 2. Easy caching of data at different levels (server, CDN network, proxy server, client device)

[0133] 3. It is easy to integrate different data storage modules, since files can be moved easily and there is a wide range of basic tools for this.

[0039] Unlike the solutions known in the prior art, where each partition has a format understandable only to a specific DBMS, in this solution each partition is stored as a separate file. This solution also proposes to combine in each file for a selected geographic area both the measures themselves that are of interest to the user and auxiliary information (area contours, points, other cartographic layers).

[0134]

[0040] Consider the traditional way of storing partitioned data, for example in PostgreSQL: Traditionally, the configuration files and data files used by a database cluster are stored together in a cluster data directory, typically called PGDATA (after the environment variable that can be used to define it). A common location for PGDATA is / var / lib / pgsql / data. Multiple clusters can exist on a single machine, managed by different server instances. The PGDATA directory contains several subdirectories and control files, as shown in the list below. In addition to these required items, the cluster configuration files postgresql.conf, pg_hba.conf, and pg_ident.conf are traditionally stored in PGDATA, although they can be placed elsewhere.

[0135] PG VERSION File containing the PostgreSQL major version number. base Subdirectory containing per-database subdirectories global Subdirectory containing cluster-wide tables such as pg_database pg_commit_ts Subdirectory containing transaction commit timestamp data. pg_clog Subdirectory containing transaction commit status data. pg dynshmem Subdirectory containing files used by the dynamic shared memory subsystem. pg logical Subdirectory containing state data for logical decoding. pg_multixact Subdirectory containing multiple transaction state data (used for shared row locks) pg_notify Subdirectory containing LISTEN / NOTIFY status data. pg replslot Subdirectory containing replication slot data. pg_serial Subdirectory containing information about committed serializable transactions. pg_snapshots Subdirectory containing exported snapshots.pg stat Subdirectory containing permanent files for the statistics subsystem. pg_stat_tmp Subdirectory containing temporary files for the statistics subsystem. pg_subtrans Subdirectory containing subtransaction status data. pg tblspc Subdirectory containing symbolic links to tablespaces. pg_twophase Subdirectory containing state files for prepared transactions. pg_xlog Subdirectory containing WAL (Write Ahead Log) files. postgresql.auto.conf File used to store configuration options set by ALTER SYSTEM. postmaster.opts File that records the command-line options the server was last started with. postmaster.pid A lock file recording the current postmaster process identifier (PID), the path to the cluster data directory, the postmaster startup timestamp, the port number, the path to the Unix domain socket directory (empty on Windows), the first valid listen_address (IP address or *, or empty if TCP is not listening), and the shared memory segment identifier (this file is missing after the server is shut down).

[0136]

[0041] For each database in the cluster, there is a subdirectory under PGDATA / base, named after the database OID in pg_database. This subdirectory is the default location for database files; in particular, its system catalogs are stored there.

[0137]

[0042] Each table and index is stored in a separate file. For normal relations, these files are named after the table's or index's filenode number, which can be found in pg class.relfilenode. But for temporary relations, the file name is of the form tBBB_FFF, where BBB is the ID of the backend that created the file and FFF is the filenode number. In either case, in addition to the main file (a / k / a main fork), each table and index has a free space map that stores information about the free space available in the relation. The free space map is stored in a file with a name consisting of the filenode number and the _fsm suffix. Tables also have a visibility map, stored in a fork with the _vm suffix, to keep track of which pages are known to be free of dead tuples. Unregistered tables and indexes have a third fork, known as the initialization fork, which is stored in the fork with the init suffix.

[0138]

[0043] Let's consider the data structure from the solution being described: the simplest structure of simple files with the same purpose. Each file contains both the geometry that fell into this tile, as well as dimensions and measures. Partitioning attributes, their number and sequence are selected depending on the problem being solved.

[0044] Below is an example of partitioning by date (y_2021-01-01), layer (fields or adm) and XYZ grid (Leaflet). Protobuf (pbf) is used as the file format:

[0139] , / у_2021-01-01 / fields / 9 / З 08_ 165.pbf

[0140] . / у_2021-01-01 / fields / 9 / З 07_ 165.pbf

[0141] . / у_2021-01-01 / fields / 9 / З 07_166. pbf

[0142] . / у_2021-01-01 / fields / 9 / 308_l 66.pbf

[0143] , / у_2021-01-01 / fields / 5 / 19_ 10.pbf

[0144] , / у_2021-01-01 / fields / 8 / 154_82.pbf

[0145] , / у_2021-01-01 / fields / 8 / 153_82.pbf

[0146]

[0045] Benefits of combining data and geometry in one file:

[0147] 1. Minimize client-server interactions

[0148] 2. Adding a new level of data granularity is easy

[0149] 3. The complexity and labor intensity of data query operations does not depend on the number of objects, dimensions and measures in one query

[0150] Example 1: The user wants to get economic indicators of the Samara region for the last several years.

[0151] In this case, the data can be partitioned as follows:

[0152] • F a l 1: “Samara region, 2020, economic indicators”

[0153] • File 2: “Samara region, 2021, economic indicators”

[0154] • File 3: “Samara region, 2022, economic indicators”

[0155] Example 2: The user wants to obtain economic indicators for one district of the Samara region for the last several years.

[0156] In this case, the data can be partitioned as follows: File 1: “Samara region, Volzhsky district, 2020, economic indicators”

[0157] • File 2: “Samara region, Volzhsky district, 2021, economic indicators”

[0158] • File 3: “Samara region, Volzhsky district, 2022, economic indicators”

[0159] As can be seen from the examples above, there is no need to recalculate previously calculated data, but only to calculate a new “scale” of data.

[0160]

[0046] Data marts in data warehouses contain a portion of pre-calculated data (data layers). In the solution described, this concept is fully implemented, and moreover, not only the data is pre-calculated, but also the server's response to the client is prepared in advance in the form of a ready-made file. The advantage of this approach is that the calculation can be performed during periods of minimal server load. All possible variants of user requests are pre-calculated, but in fact only the changed files are replaced. However, if the server part of the method implementation can determine the set of files affected by the recalculation, then only these files can be recalculated and replaced. For example, if the economic indicators of one region of the Russian Federation have changed, then only the files affecting this region will be recalculated, as well as all files of a higher hierarchy level (aggregates for the entire Russian Federation). At the same time, files related to other regions will be recalculated and will not be affected.

[0161]

[0047] At the same time, the method does not exclude the parallel use of traditional DBMS for storing pre-calculated data on the server based on tile identifiers, in particular, for partitioning, storing auxiliary information, calculating aggregates. The optimization of the main user function - viewing data - is essential in this method.

[0048] Tile identifiers can be obtained: either from input requirements for calculation; or based on the received geodata, or initially all possible values ​​are taken.

[0162]

[0049] The file identifier encoding system is known to both the server and the client and allows the identifier of the requested data object to be easily obtained.

[0163]

[0050] The coding system can be either one of the standard ones or proprietary for each client.

[0164] As an example, let's consider:

[0165] • XYZ grid from Leaflet

[0166] For example, to visualize a map of our planet, you can use a cylindrical projection (if distortions of this projection are acceptable) and cutting into tiles of the same size (in pixels).

[0167] At maximum distance (Zoom = 0) the entire map fits into one tile: in Fig.2.

[0168] At each subsequent zoom level, the number of tiles increases by 2 times (relative to the previous one) along the x and y axes.

[0169] Zoom = 1 : in Fig.2;

[0170] Zoom = 2: in Fig.3;

[0171] Zoom = 13: in Fig. 4.

[0172]

[0051] In a particular embodiment, the user selects the geographic area he needs by generating the identifier of the desired file according to a previously known formula. The user then accesses the specific file.

[0173]

[0052] In a user application, such as a browser, the identifiers of the files required for loading are determined based on the coding system known from the previous point and the user's current actions, such as the selected display area. Advantage: the formula is known in advance, it is simple, so this operation does not require large computing capacities.

[0174]

[0053] The user device, based on the received list of file identifiers, loads them in a standard way for the user OS. Advantages:

[0175] • no need to implement file loading - this functionality is standard and has already been implemented in the user's OS;

[0176] • parallelization can be used if there is support at the OS and protocol level, using various protocols (HTTP, FTP, S3);

[0177] • it is possible to apply caching at various levels (user device, local network (proxy server), server);

[0178] • local caching of data on the user side is easily implemented, for example, when implementing a mobile application, since it is necessary to save files.

[0179]

[0054] When using the main user function of the system (data viewing), when changing data in it, the user device downloads only the changes, not all the data. There are several known ways to implement this step, for example, using the E-Tag header of the HTTP protocol, parameterization of the rules for determining file identifiers. Advantages:

[0180] • minimization of the volume of transferred data;

[0181] • availability of existing implementations of this step;

[0182] • unlimited horizontal scaling of the system, for example, you can simply copy the same files to N servers and perform balancing using the round robin method, which eliminates the need for complex management of the container architecture; • high speed of the server part, since there are no server calculations during the user's request (all data is pre-calculated);

[0183] • high speed of the server part, since the user downloads mainly the necessary data;

[0184] • convenient management of data granularity.

[0185]

[0055] In general terms (see Fig. 5), the system for forming partitioned data marts containing geodata and their use during the operation of the data warehouse (500) contains one or more processors (501) united by a common information exchange bus, memory means such as RAM (502) and ROM (503), and input / output interfaces (504).

[0186]

[0056] The processor (501) (or several processors, a multi-core processor, etc.) can be selected from a range of devices that are widely used at present, for example, from manufacturers such as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, etc. The processor or one of the processors used in the system (500) must also include a graphics processor, for example, an NVIDIA GPU with a software model compatible with CUD A, or Graphcore, the type of which is also suitable for the full or partial implementation of the method, and can also be used for training and applying machine learning models in various information systems.

[0187]

[0057] RAM (502) is a random access memory and is intended for storing machine-readable instructions executable by the processor (501) for performing the necessary operations for logical data processing. RAM (502), as a rule, contains executable instructions of the operating system and the corresponding software components (applications, software modules, etc.). In this case, the available memory capacity of the graphics card or graphics processor can act as RAM (502).

[0058] ROM (503) is one or more permanent data storage devices, for example, a hard disk (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0188]

[0059] To organize the operation of the components of the device (500) and to organize the operation of external connected devices, various types of I / O interfaces (504) are used. The selection of the corresponding interfaces depends on the specific design of the computing device, which may be, without limitation: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0189]

[0060] To ensure user interaction with the device (500), various I / O information means (505) are used, for example, a keyboard, a display (monitor), a touch display, a touchpad, a joystick, a mouse, a light pen, a stylus, a touch panel, a trackball, speakers, a microphone, augmented reality means, optical sensors, a tablet, light indicators, a projector, a camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.

[0190]

[0061] The network interaction means (506) provides data transmission via an internal or external computer network, for example, an Intranet, the Internet, a LAN, etc. One or more means (506) may be, but are not limited to: an Ethernet card, a GSM modem, a GPRS modem, an LTE modem, a 5G modem, a satellite communication module, an NFC module, a Bluetooth and / or BLE module, a Wi-Fi module, etc.

[0191]

[0062] The specific selection of elements of the device (500) for implementing various software and hardware architectural solutions may vary while maintaining the required functionality provided. In particular, such an implementation may be performed using electronic components used to create digital integrated circuits. Without limitation, microcircuits may be used whose operating logic is determined during manufacture, or programmable logic integrated circuits (PLIS), whose operating logic is specified by programming. For programming, programmers and debugging environments are used that allow the desired structure of the digital device to be specified in the form of a basic electrical circuit or a program in special hardware description languages: Verilog, VHDL, AHDL, etc.An alternative to FPGAs are: programmable logic controllers (PLCs), basic matrix crystals (BMCs), which require a factory production process for programming; ASICs — specialized custom large-scale integrated circuits (LSIs), which are significantly more expensive for small-scale and single-unit production. Thus, implementation can be achieved by standard means based on classical principles of implementing the fundamentals of computer technology.

[0063] The submitted application materials disclose preferred examples of implementing the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims

FORMULA 1. A method for creating partitioned data marts containing geodata and using them in the process of operating a data warehouse, implemented using a processor and a data storage device, including the following steps: • receive input data including: geodata, a list of numerical values ​​from the minimum to the maximum value of the scale level of the generated data marts and a dataset with arbitrary attributes for generating a data mart; • combine the obtained geodata and dataset; • based on the result of the merge carried out in the previous step, tiles and their identifiers are formed for all zoom levels of the generated data marts as follows: o for the current zoom level, in a loop by tile identifiers, all the data available for this tile in the merged dataset are obtained; o the tiles are converted into a format for processing by the data mart and stored in the memory of the data storage device. • use the tiles obtained during the operation of the data warehouse to form partitioned data marts containing geodata in the following way: o select the required zoom level; o automatically determine all tile identifiers at the current zoom level; o request tiles according to the defined identifiers, create and visualize the corresponding data marts.

2. The method according to paragraph 1, characterized in that when forming tiles and their identifiers, additional calculations are performed on the data, allowing the attributes to be changed.

3. The method according to paragraph 1, characterized in that the selection of the scale level is carried out through preset values.

4. The method according to paragraph 1, characterized in that the selection of the zoom level is carried out dynamically.

5. A system for the formation of partitioned data marts containing geodata and their use in the process of operating a data warehouse, comprising: o at least one data processing device; o at least one data storage device; o at least one program, where one or more programs are stored on one or more data storage devices and executed on one or more data processing devices, wherein one or more programs ensure the execution of the following steps: • receive input data including: geodata, a list of numerical values ​​from the minimum to the maximum value of the scale level of the generated data marts and a dataset with arbitrary attributes for generating a data mart; • combine the obtained geodata and dataset; • based on the result of the merge carried out in the previous step, tiles and their identifiers are formed for all zoom levels of the generated data marts as follows: o for the current zoom level, in a loop by tile identifiers, all the data available for this tile in the merged dataset are obtained; o convert tiles into a format suitable for processing by the data mart and store them in the memory of the data storage device. • use the tiles obtained during the operation of the data warehouse to form partitioned data marts containing geodata in the following way: o select the required zoom level; o automatically determine all tile identifiers at the current zoom level; o request tiles according to the defined identifiers, create and visualize the corresponding data marts.

6. The system according to I.5, characterized by the fact that when forming tiles and their identifiers, additional calculations are performed on the data, allowing the attributes to be changed.

7. The system according to paragraph 5, characterized in that the selection of the scale level is carried out through preset values.

8. The system according to paragraph 5, characterized in that the selection of the zoom level is carried out dynamically.

Images