Vector data slice processing method and device in big data set

Abstract

The invention provides a vector data slicing processing method and device in a big data set, and belongs to the technical field of map data processing.The vector data slicing processing method in the big data set comprises the steps that vector data in the big data set is preprocessed; dividing the preprocessed big data set into a top-layer grid comprising a plurality of areas, and subdividing the top-layer grid to form a plurality of sub-grids; dynamically adjusting the granularity of grid division according to the interaction behavior of the user and the map zooming level; data processing in each grid unit after dynamic adjustment is used as an independent task, and the independent task is allocated to different computing nodes or threads for independent processing; and combining the processed data in each grid unit to generate vector slice data and outputting the vector slice data. The vector data slice processing method in the big data set is better in performance, higher in processing speed, lower in resource consumption, higher in precision and better in expansibility.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a method and device for processing vector data slices in a large data set. Background Art

[0002] With the rapid growth of geospatial data, traditional vector data slicing technology faces many challenges when processing large-scale data, such as low data processing efficiency, imperfect caching mechanism, and insufficient load balancing.

[0003] MVT (Mapbox VectorTiles) technology has been widely used in the GIS (Geographic Information System) field due to its efficient data compression and transmission capabilities. However, in large data volume scenarios, the existing MVT implementation still has performance bottlenecks and needs to be further optimized to improve its processing capabilities and response speed. Summary of the invention

[0004] In view of this, the present invention provides a method and device for processing vector data slicing in a large data set, which is used to solve the problems that the current vector data slicing algorithm has high computational complexity when processing large-scale data sets, resulting in a long slicing process that affects the overall efficiency, requires a large amount of computing resources, loses vector data accuracy, has poor scalability, and has insufficient rendering and slicing quality.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides a method for processing vector data slices in a large data set, the method comprising:

[0006] Preprocessing of vector data in large data sets;

[0007] Dividing the preprocessed large data set into a top-level grid containing a plurality of regions, and subdividing the top-level grid to form a plurality of sub-grids;

[0008] Dynamically adjust the granularity of grid division according to user interaction behavior and map zoom level;

[0009] The data processing in each dynamically adjusted grid unit is treated as an independent task and assigned to different computing nodes or threads for independent processing;

[0010] The processed data in each grid unit is merged to generate vector slice data and output it.

[0011] Optionally, the step of dynamically adjusting the granularity of grid division according to the user's interactive behavior and the map zoom level includes:

[0012] The density and distribution of vector data in the current map view are analyzed in real time based on the user's interactive behavior, and the granularity and slicing strategy of grid division are dynamically adjusted.

[0013] Optionally, the step of dynamically adjusting the granularity of grid division according to the user's interactive behavior and the map zoom level includes:

[0014] At high zoom levels, a strategy of progressively refining the mesh granularity is adopted to gradually increase the detail and resolution of the vector data.

[0015] Optionally, the step of analyzing the density and distribution of vector data in the current map view in real time according to the user's interactive behavior and dynamically adjusting the granularity of grid division and slicing strategy includes.

[0016] When the user's interactive behavior is fast zooming, high-density vector data in the center area of ​​the map view is processed first, and the processing of vector data in the edge area is reduced.

[0017] Optionally, before the step of merging the processed data in each of the grid units to generate and output vector slice data, the method further includes:

[0018] Performing boundary point detection on the vector data on the boundary of each grid unit, and marking the boundary point of each grid unit;

[0019] Boundary points of adjacent grid units are fused.

[0020] Optionally, the step of preprocessing the vector data in the large data set includes:

[0021] Building a spatial index for the vector data in the large data set;

[0022] The vector data is normalized to convert the vector data of different scales into a unified coordinate system.

[0023] In a second aspect, the present invention further provides a device for processing vector data slices in a large data set, the device comprising:

[0024] A preprocessing module is used to preprocess the vector data in the large data set;

[0025] A partitioning module is used to partition the preprocessed large data set into a top-level grid containing a plurality of regions, and to subdivide the top-level grid to form a plurality of sub-grids;

[0026] Dynamic adjustment module, used to dynamically adjust the granularity of grid division according to user interaction behavior and map zoom level;

[0027] An allocation module is used to allocate the data processing in each grid unit after dynamic adjustment as an independent task to different computing nodes or threads for independent processing;

[0028] The output module is used to merge the processed data in each grid unit, generate vector slice data and output it.

[0029] Optionally, the dynamic adjustment module includes:

[0030] The first dynamic adjustment unit is used to analyze the density and distribution of vector data in the current map view in real time according to the user's interactive behavior, and dynamically adjust the granularity and slicing strategy of the grid division.

[0031] Optionally, the dynamic adjustment module includes.

[0032] The second dynamic adjustment unit is used to gradually increase the details and resolution of the vector data by adopting a strategy of progressively refining the granularity of the grid at a high zoom level.

[0033] Optionally, the root first dynamic adjustment unit includes.

[0034] The dynamic adjustment subunit is used to give priority to processing high-density vector data in the central area of ​​the map view and reduce the processing of vector data in the edge area when the user's interactive behavior is fast zooming.

[0035] Optionally, the device further comprises:

[0036] A marking module, used to perform boundary point detection on the vector data on the boundary of each grid unit and mark the boundary point of each grid unit;

[0037] The fusion module is used to fuse the boundary points of adjacent grid units.

[0038] Optionally, the preprocessing module includes:

[0039] A construction unit, used to construct a spatial index for the vector data in the large data set;

[0040] The normalization processing unit is used to perform normalization processing on the vector data and convert the vector data of different scales into a unified coordinate system.

[0041] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in any of the above-mentioned vector data slicing processing methods in a large data set are implemented.

[0042] The beneficial effects of the above technical solution of the present invention are as follows:

[0043] In the embodiment of the present invention, by multi-level grid division and dynamic adjustment of the grid granularity, and the use of parallel processing technology, redundant calculation and storage can be reduced, system resource consumption can be reduced, the slicing efficiency of large-scale data can be accelerated, the processing time can be reduced, the high precision of vector data in the slicing process can be ensured, more detailed information can be retained, and it has better scalability, reduces the delay of data transmission and processing, ensures that the front-end rendering is not stuck, and provides a smooth user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of a method for processing vector data slices in a large data set in Embodiment 1 of the present invention;

[0045] Figure 2 It is a structural schematic diagram of a vector data slice processing device in a large data set in the second embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0047] With the rapid growth of geospatial data, traditional vector data slicing technology faces many challenges when processing large-scale data, such as low data processing efficiency, imperfect caching mechanism, and insufficient load balancing.

[0048] MVT technology has been widely used in the GIS field due to its efficient data compression and transmission capabilities. However, in large data volume scenarios, the implementation of existing MVT technology still has performance bottlenecks and needs further optimization to improve its processing capabilities and response speed.

[0049] The inventors of the present application have conducted a detailed analysis of the existing MVT vector data slicing technology and found that it has some problems, which mainly include:

[0050] 1. Performance bottleneck:

[0051] Existing slicing algorithms have high computational complexity when processing large-scale data sets, which causes the slicing process to take a long time and affects the overall efficiency. Especially when the data set contains tens of thousands or hundreds of thousands of vector data, traditional algorithms are difficult to process efficiently and show obvious performance bottlenecks.

[0052] 2. Resource consumption:

[0053] In the big data environment, existing algorithms consume large amounts of memory and CPU resources. In order to ensure the quality and speed of slicing, a large amount of computing resources are usually required, which makes it difficult to achieve efficient operation requirements for resource-constrained devices (such as mobile devices).

[0054] 3. Loss of precision:

[0055] Some existing algorithms may cause loss of precision in vector data during the slicing process. Since data needs to be simplified to reduce the amount of calculation, some detailed information may be lost, affecting the final rendering effect. For example, small geographic features may be ignored or distorted during the slicing process.

[0056] 4. Poor scalability:

[0057] Existing slicing algorithms are difficult to adapt to the needs of different application scenarios, especially in applications with high real-time requirements. As the amount of data and user needs increase, existing algorithms are difficult to effectively expand, resulting in a decrease in system response speed and a poor user experience.

[0058] 5. Pyramid Slicing Limitations:

[0059] Although the traditional pyramid slicing scheme has certain advantages in multi-level zooming, it has shortcomings in rendering efficiency and slicing quality in practical applications. When users frequently switch between different zoom levels, the preprocessing and storage requirements of the pyramid slices are high, which increases the system burden.

[0060] Therefore, please refer to Figure 1 , Figure 1 A flowchart of a method for processing vector data slices in a large data set provided in the first embodiment of the present invention, the method comprising the following steps:

[0061] Step 11: Preprocess the vector data in the large data set.

[0062] In this embodiment, the method can be used to process large-scale and complex data sets. The purpose of preprocessing the vector data on the large data set is to optimize the data structure in the subsequent slicing process, improve the processing efficiency, and ensure the integrity and accuracy of the data. The preprocessing process may include spatial index construction and data normalization processing.

[0063] In an embodiment of the present invention, the step of preprocessing the vector data in the large data set includes:

[0064] Building a spatial index for the vector data in the large data set;

[0065] The vector data is normalized to convert the vector data of different scales into a unified coordinate system.

[0066] The spatial index can be constructed using an R-tree or a quadtree, so that data points can be quickly located and retrieved. In this embodiment, when data of different scales are converted into a unified coordinate system, the coordinate point (x, y) is mapped to (x', y') in the normalized coordinate system, which can be specifically expressed by the following formula:

[0067]

[0068] Step 12: Divide the preprocessed large data set into a top-level grid containing a plurality of regions, and subdivide the top-level grid to form a plurality of sub-grids.

[0069] In this step, a multi-layer grid partitioning method is used to process large-scale vector data in blocks according to predefined grids. The data of each grid unit is processed independently to reduce the amount of vector data processed at a single time. First, the entire preprocessed data set is divided into several top-level grids of larger areas, and the boundaries of these grids ensure that the entire data set is covered. After that, each top-level grid is further divided to form multiple sub-grids, each of which processes smaller data blocks to improve processing efficiency and fine-grained control.

[0070] Top grid G i，j is defined as follows:

[0071] G i,j ={(x, y)∈D|x∈[x i , x i+1 ), y∈[y j ,y j+1 )};

[0072] Subgrid G i，j，k，l Definition:

[0073] G i，j，k，l ={(x, y)∈G i，j |x∈[x k , x k+1 ), y∈[y l ,y l+1 )}.

[0074] Step 13: Dynamically adjust the granularity of grid division according to the user's interactive behavior and the map zoom level.

[0075] In this embodiment, while adopting multi-level grid division, the granularity of grid division is dynamically adjusted according to the map zoom level based on the user's interactive behavior (such as zooming and panning) and combined with a dynamic level optimization strategy to ensure the optimal data volume and rendering efficiency at different levels.

[0076] Step 14: The data processing in each dynamically adjusted grid unit is treated as an independent task and assigned to different computing nodes or threads for independent processing.

[0077] In this step, multithreading or distributed computing framework is used to achieve parallel processing and improve slicing efficiency. The data processing in each grid unit is treated as an independent processing task and assigned to parallel multithreading for independent processing, or different computing nodes under the distributed computing framework, so as to achieve parallel processing of grid unit data.

[0078] In this embodiment, the grid unit may be a top-level grid unit or a sub-grid unit. The processing of the computing node or computing thread includes data filtering, boundary processing, and the like.

[0079] The formula for parallel processing is expressed as:

[0080] Map(k1,v1)→〈k2,v2〉

[0081] Reduce(k2, <v2〉)→〈k3,v3〉。

[0082] Step 15: Merge the processed data in each grid unit to generate vector slice data and output it.

[0083] After processing the data in the grid cells, the processed data are merged to generate final vector slice data and output. Specifically, the final vector slice data can be converted into MVT format and output to a specified location.

[0084] In this embodiment, the above steps are expressed as:

[0085]

[0086] The vector data slicing processing method in a large data set provided by an embodiment of the present invention can reduce redundant calculations and storage, reduce system resource consumption, speed up the slicing efficiency of large-scale data, reduce processing time, ensure high precision of vector data in the slicing process, retain more detailed information, have better scalability, reduce data transmission and processing delays, ensure that front-end rendering is not stuck, and provide a smooth user experience.

[0087] The following example illustrates the vector data slicing processing method in the above large data set.

[0088] In one optional specific implementation manner, the step of dynamically adjusting the granularity of grid division according to the user's interactive behavior and the map zoom level includes:

[0089] The density and distribution of vector data in the current map view are analyzed in real time based on the user's interactive behavior, and the granularity and slicing strategy of grid division are dynamically adjusted.

[0090] In this embodiment, the user's interactive behavior can be operations such as zooming and panning the map. By analyzing the density and distribution of vector data in the current map view in real time according to the user's interactive behavior and flexibly adjusting the slicing strategy, more detailed information can be retained.

[0091] In an embodiment of the present invention, the step of analyzing the density and distribution of vector data in the current map view in real time according to the user's interactive behavior and dynamically adjusting the granularity and slicing strategy of the grid division includes:

[0092] When the user's interactive behavior is fast zooming, high-density vector data in the center area of ​​the map view is processed first, and the processing of vector data in the edge area is reduced.

[0093] In this step, when the user's interactive behavior is fast zooming, by giving priority to processing high-density vector data in the center area of ​​the map view and reducing the processing of vector data in the edge area, the processing efficiency can be improved, the rendering efficiency can be improved, and the front-end rendering can be ensured without lag, so that users can quickly browse the map details in the center area of ​​the map view, thereby providing a smooth user experience. The processing formula can be expressed as:

[0094] D active ={(x, y)∈D|x∈[x min , x max ],y∈[y min ,y max ]};

[0095] Among them, D active is the active data range in the current view, x min , x max ,y min ,y max is the view bounds.

[0096] In the embodiment of the present invention, the step of dynamically adjusting the granularity of grid division according to the user's interactive behavior and the map zoom level includes:

[0097] At high zoom levels, a strategy of progressively refining the mesh granularity is adopted to gradually increase the detail and resolution of the vector data.

[0098] In this embodiment, at a high zoom level, a progressive refinement strategy is used to gradually increase the details and resolution of the vector data, avoiding performance bottlenecks caused by loading a large amount of vector data at one time. In addition, through the progressive refinement strategy, small geographic features (such as small roads and building boundaries) can be fully preserved in the slices, and the presentation effect is more refined. The processing formula can be expressed as:

[0099] G refine =G i，j，k，l |if z>z threshold ;

[0100] Among them, z threshold is the refinement threshold, G refine is the refined mesh data.

[0101] In the embodiment of the present invention, before the step of merging the processed data in each grid unit to generate vector slice data and outputting it, the method further includes:

[0102] Performing boundary point detection on the vector data on the boundary of each grid unit, and marking the boundary point of each grid unit;

[0103] Boundary points of adjacent grid units are fused.

[0104] In this step, the boundary points of each grid unit are detected and marked, and the boundary points of adjacent grid units are fused to ensure the continuity and consistency of the vector data. By performing special processing on the vector data on the grid unit boundary, the vector data at the boundary can be seamlessly connected, ensuring the high accuracy of the vector data during the slicing process. The processing formula can be expressed as:

[0105] B ij ={(x,y)∈Q i ∩Q j |(x,y) is a boundary point}.

[0106] In summary, the present invention realizes an efficient vector data slicing processing method, which can quickly and accurately perform vector data slicing processing in a big data environment, meeting the needs of various application scenarios. Specifically, the vector data slicing processing method in a large data set of the present invention achieves significant improvements and optimizations in the following aspects:

[0107] (1) Processing speed is greatly improved:

[0108] The algorithm of the present invention is used to perform slicing on a large data set containing millions of vector data. Compared with the existing MVT slicing algorithm, the processing speed is increased by about 50%. Through multi-level grid division and parallel processing technology, the slicing process is more efficient and the processing time is significantly reduced.

[0109] (2) Resource consumption is significantly reduced:

[0110] By optimizing the algorithm's data structure and calculation method, the memory and CPU usage are reduced and resource utilization efficiency is improved. Memory usage is reduced by about 30% and CPU usage is reduced by about 20%, making it possible to run efficiently even on resource-constrained devices.

[0111] (3) Slicing accuracy is significantly improved:

[0112] Adopt precise boundary processing and data fusion technology to ensure high accuracy of vector data during slicing. Dynamically adjust the grid division granularity, flexibly adjust the slicing strategy according to data density and zoom level, retain more detailed information, without any data loss or accuracy loss.

[0113] For example, at high zoom levels (such as 1:1000 scale), small geographic features (such as small roads and building boundaries) are fully preserved in the slices, and the presentation effect is more refined.

[0114] (4) Enhanced system scalability:

[0115] Design a highly scalable algorithm architecture that can adapt to data sets of different sizes and diverse application scenarios. Support multi-threading and distributed computing, improve the system's concurrent processing capabilities, and meet the needs of applications with high real-time requirements.

[0116] (5) Front-end rendering optimization:

[0117] Introducing dynamic hierarchical optimization strategies, flexibly adjusting mesh division and slicing strategies at different zoom levels to improve rendering efficiency. Combining multi-level cache management technology to reduce data transmission and processing delays, ensure that front-end rendering is not stuck, and provide a smooth user experience.

[0118] For example, in an online map service, when a user quickly zooms in and out of a map, the algorithm of the present invention can respond to user requests in milliseconds and display high-quality map slices in real time, significantly improving the user experience.

[0119] See also Figure 2 , Figure 2 2 is a schematic diagram of a vector data slice processing device in a data set provided by Embodiment 2 of the present invention. The device 20 includes:

[0120] A preprocessing module 21 is used to preprocess the vector data in the large data set;

[0121] A division module 22 is used to divide the preprocessed large data set into a top-level grid containing a plurality of regions, and subdivide the top-level grid to form a plurality of sub-grids;

[0122] A dynamic adjustment module 23, used to dynamically adjust the granularity of grid division according to the user's interactive behavior and the map zoom level;

[0123] An allocation module 24 is used to allocate the data processing in each grid unit after dynamic adjustment as an independent task to different computing nodes or threads for independent processing;

[0124] The output module 25 is used to merge the processed data in each grid unit, generate vector slice data and output it.

[0125] Optionally, the dynamic adjustment module 23 includes:

[0126] The first dynamic adjustment unit is used to analyze the density and distribution of vector data in the current map view in real time according to the user's interactive behavior, and dynamically adjust the granularity and slicing strategy of the grid division.

[0127] Optionally, the dynamic adjustment module 23 includes.

[0128] The second dynamic adjustment unit is used to gradually increase the details and resolution of the vector data by adopting a strategy of progressively refining the granularity of the grid at a high zoom level.

[0129] Optionally, the root first dynamic adjustment unit includes.

[0130] The dynamic adjustment subunit is used to give priority to processing high-density vector data in the central area of ​​the map view and reduce the processing of vector data in the edge area when the user's interactive behavior is fast zooming.

[0131] Optionally, the device 20 further includes:

[0132] A marking module, used to perform boundary point detection on the vector data on the boundary of each grid unit and mark the boundary point of each grid unit;

[0133] The fusion module is used to fuse the boundary points of adjacent grid units.

[0134] Optionally, the preprocessing module 21 includes:

[0135] A construction unit, used to construct a spatial index for the vector data in the large data set;

[0136] The normalization processing unit is used to perform normalization processing on the vector data and convert the vector data of different scales into a unified coordinate system.

[0137] In the embodiment of the present invention, by multi-level grid division and dynamic adjustment of the grid granularity, and the use of parallel processing technology, redundant calculation and storage can be reduced, system resource consumption can be reduced, the slicing efficiency of large-scale data can be accelerated, the processing time can be reduced, the high precision of vector data in the slicing process can be ensured, more detailed information can be retained, and it has better scalability, reduces the delay of data transmission and processing, ensures that the front-end rendering is not stuck, and provides a smooth user experience.

[0138] The embodiment of the present invention is a product embodiment corresponding to the above-mentioned method embodiment 1, so it will not be described in detail here. Please refer to the above-mentioned embodiment 1 for details.

[0139] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for processing vector data slices in a large data set, characterized in that: include: Preprocessing of vector data in large data sets; Dividing the preprocessed large data set into a top-level grid containing a plurality of regions, and subdividing the top-level grid to form a plurality of sub-grids; Dynamically adjust the granularity of grid division according to user interaction behavior and map zoom level; The data processing in each dynamically adjusted grid unit is treated as an independent task and assigned to different computing nodes or threads for independent processing; The processed data in each grid unit is merged to generate vector slice data and output it.

2. The method according to claim 1, characterized in that The step of dynamically adjusting the granularity of grid division according to the user's interactive behavior and the map zoom level includes: The density and distribution of vector data in the current map view are analyzed in real time based on the user's interactive behavior, and the granularity and slicing strategy of grid division are dynamically adjusted.

3. The method according to claim 1 or 2, characterized in that: The step of dynamically adjusting the granularity of grid division according to the user's interactive behavior and the map zoom level includes: At high zoom levels, a strategy of progressively refining the mesh granularity is adopted to gradually increase the detail and resolution of the vector data.

4. The method according to claim 2, characterized in that: The steps of analyzing the density and distribution of vector data in the current map view in real time according to the user's interactive behavior and dynamically adjusting the granularity and slicing strategy of grid division include: When the user's interactive behavior is fast zooming, high-density vector data in the center area of ​​the map view is processed first, and the processing of vector data in the edge area is reduced.

5. The method according to claim 1, characterized in that: Before the step of merging the processed data in each grid unit to generate vector slice data and outputting the vector slice data, the method further includes: Performing boundary point detection on the vector data on the boundary of each grid unit, and marking the boundary point of each grid unit; Boundary points of adjacent grid units are fused.

6. The method according to claim 1, characterized in that The step of preprocessing the vector data in the large data set comprises: Building a spatial index for the vector data in the large data set; The vector data is normalized to convert the vector data of different scales into a unified coordinate system.

7. A vector data slice processing device in a large data set, characterized in that: include: A preprocessing module is used to preprocess the vector data in the large data set; A partitioning module is used to partition the preprocessed large data set into a top-level grid containing a plurality of regions, and to subdivide the top-level grid to form a plurality of sub-grids; Dynamic adjustment module, used to dynamically adjust the granularity of grid division according to user interaction behavior and map zoom level; An allocation module is used to allocate the data processing in each grid unit after dynamic adjustment as an independent task to different computing nodes or threads for independent processing; The output module is used to merge the processed data in each grid unit, generate vector slice data and output it.

8. The device according to claim 7, characterized in that The dynamic adjustment module comprises: The first dynamic adjustment unit is used to analyze the density and distribution of vector data in the current map view in real time according to the user's interactive behavior, and dynamically adjust the granularity and slicing strategy of the grid division.

9. The device according to claim 7 or 8, characterized in that The dynamic adjustment module includes. The second dynamic adjustment unit is used to gradually increase the details and resolution of the vector data by adopting a strategy of progressively refining the granularity of the grid at a high zoom level.

10. The device according to claim 8, characterized in that The root first dynamic adjustment unit includes. The dynamic adjustment subunit is used to give priority to processing high-density vector data in the central area of ​​the map view and reduce the processing of vector data in the edge area when the user's interactive behavior is fast zooming.

Images