A High-Performance and Fast Visualization Rendering Optimization Method and Device for Massive Data

Abstract

The present invention discloses a high-performance method and device for rapid visualization rendering optimization of massive data, belonging to the technical field of data processing and visualization. The method includes generating a clustering data set with grid index numbers at the server end according to a request from the browser end, and compressing the data set; transmitting the compressed data to the browser end through the network, and the browser end decompresses the received data and restores it to a data set with grid index numbers; performing density analysis on the data set after decompression, calculating the density weight of each grid, and downsampling the data of each grid; mapping the reference circle divided by the grid according to the rendering visible area in the browser, and performing hierarchical rendering on the mapped grid in the browser based on the grid density weight. The present invention improves the rendering efficiency, shortens the first rendering time, and realizes rapid, real-time, and accurate visualization rendering of massive data.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and visualization, and specifically relates to a high-performance mass data fast visualization rendering optimization method and device. Background Art

[0002] With the continuous development and popularization of information technology, the application of big data has penetrated into all walks of life and has become an important force in promoting social progress and economic development. However, the visualization and rendering of massive data has always been a problem that has plagued the development of the industry. The traditional data visualization method is to pre-process the data on the server side, process it into the data format defined by the interface, and respond to the browser side for visualization rendering. This method can achieve fast data rendering when the data scale is small, but when the data scale increases, the rendering speed will slow down significantly.

[0003] Many scholars have conducted research on this issue and optimized traditional data processing and rendering methods. These optimizations mainly include adding data screening and deduplication steps during data preprocessing on the server side, and achieving rendering optimization through interpolation and other technologies on the browser side. The optimized methods can improve the processing and optimization of a certain data scale.

[0004] Based on the current research of scholars, the processing and optimization of data scale can be improved to a certain extent. However, when the data scale continues to increase, the overall rendering speed is often slow and the effect is poor due to slow preprocessing speed, large amount of data transmission, and insufficient memory or congestion of the browser when rendering massive data. There are even problems such as freeze, delay and crash of the browser, which cannot meet the user's requirements for real-time and accuracy, and greatly reduces the user experience. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a high-performance optimization method and device for rapid visualization rendering of massive data, which optimizes the visualization rendering process of massive data from three aspects: data processing on the server side, data transmission, and data rendering on the browser side, and ultimately solves the problems of slow visualization rendering speed and poor effect of large-scale data, and realizes rapid, real-time and accurate visualization rendering of massive data on the browser side.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A high-performance mass data fast visualization rendering optimization method, the method comprising:

[0008] Step 1: According to the request of the browser, the server screens, removes duplicates and dynamically divides the data in the database into grids, generates a clustered data set with a grid index number, and compresses the data set;

[0009] Step 2: Transmit the compressed data to the browser side through the network. The browser side decompresses the received data and restores it to a data set with grid index numbers.

[0010] Step 3: Perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight.

[0011] Step 4: Map the reference circle of grid division according to the rendering visible area in the browser, and perform hierarchical rendering of the mapped grid in the browser based on the grid density weight.

[0012] On the other hand, the present invention provides a high-performance massive data fast visualization rendering optimization device, including:

[0013] A preprocessing module, configured to screen, deduplicate, and perform dynamic grid division on the data in the database according to the request of the browser side, generate a clustering data set with grid index numbers, and compress the data set.

[0014] A decompression module, configured to transmit the compressed data to the browser side through the network. The browser side decompresses the received data and restores it to a data set with grid index numbers.

[0015] A sampling module, configured to perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight.

[0016] A rendering module, configured to map the reference circle of grid division according to the rendering visible area in the browser, and perform hierarchical rendering of the mapped grid in the browser based on the grid density weight.

[0017] In a third aspect, the present invention provides an electronic device, including: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing high-performance massive data fast visualization rendering optimization method.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor can implement the foregoing high-performance massive data fast visualization rendering optimization method.

[0019] The beneficial effects of the present invention are as follows:

[0020] Introduce the grid division of two-dimensional data. The grid division based on the reference circle can effectively avoid the omission of boundary data and ensure the integrity of the data.

[0021] Perform downsampling on each grid based on density weights. While reducing the amount of rendered data, ensure that the retained data can accurately reflect the data distribution characteristics and trends, and improve the accuracy of visualization.

[0022] Layer the data based on grid density weights, and achieve hierarchical rendering of the data by monitoring user behavior, thereby improving the rendering efficiency. Description of the Drawings

[0023] Figure 1 It is a flowchart of a high-performance massive data fast visualization rendering optimization method of the present invention. Detailed Embodiment

[0024] The present invention will be further described below in conjunction with the drawings and embodiments.

[0025] The present invention provides a high-performance massive data fast visualization rendering optimization method. It introduces the concept of grid data commonly used in geographic information systems, dynamically divides large-scale data into grids on the server side, performs operations such as data format conversion and data assembly based on the divided grids to obtain response data that can respond to browser requests, and compresses the response data before transmission. This can reduce the transmission time overhead caused by overly large data files. Receive the compressed response data at the browser side and decompress it, and perform downsampling based on data density weights. While reducing the amount of data, ensure that the data for rendering is effective and accurate. Finally, adopt hierarchical rendering during the rendering process, and perform hierarchical rendering in the browser according to the rendering visible area and grid index number in the browser to accelerate the rendering efficiency, and finally achieve fast visualization rendering of large-scale data at the browser side.

[0026] As Figure 1 shown, it is a flowchart of a high-performance massive data fast visualization rendering optimization method of the present invention. Assume that the amount of data for visualization rendering is in the millions, and it is discontinuous discrete values. The network state is good during the data transmission process, ignoring network latency, and the CPU is used for data volume color calculation or fusion processing during the rendering process. The method specifically includes:

[0027] Step 1: According to the request from the browser side, the server side filters, de-duplicates, and dynamically divides the grids of the data in the database, generates a clustering data set with grid index numbers, and compresses the data set; specifically:

[0028] Step 1.1: The server - side parses the conditions of the request sent by the browser, extracts the condition information to generate a database query statement, filters the data that meets the conditions in the database through the query statement, and at the same time removes duplicates of the filtered data using a Bloom filter to eliminate the dirty data generated by unreliable data sources, obtaining preliminary response data;

[0029] Step 1.2: Dynamically divide the obtained preliminary response data according to certain rules into data dynamic grids. Divide all data into multiple grids according to the corresponding characteristic quantities to achieve data clustering. Specifically:

[0030] Step 1.2.1: Dynamically calculate the center point of the grid division based on the coordinate point information of all data as the reference point, and determine the circular space range centered on the reference point as the reference circle; Since the obtained preliminary response data needs to be displayed on the interface, there is two - dimensional coordinate point information. The rule for dynamic grid division is to dynamically obtain the center point for grid division based on the coordinate point information of all data as the reference point:

[0031] ,

[0032] where, are the X - axis and Y - axis coordinates of the reference point, is the maximum X - axis coordinate in the preliminary response data, is the minimum X - axis coordinate in the preliminary response data, is the maximum Y - axis coordinate in the preliminary response data, is the minimum Y - axis coordinate in the preliminary response data.

[0033] Determine the circular space range for grid division (hereinafter referred to as the reference circle) based on the reference point:

[0034] ,

[0035] where, is the radius of the reference circle.

[0036] Step 1.2.2: Select a reference radius, calculate the median of the distances from the coordinate points on the reference radius to the center of the reference circle as the division boundary of the grid, and divide the reference circle into two parts with the division boundary as the radius; Select as the reference radius, and calculate the median of the distances between the coordinate points on the reference radius and the center of the reference circle (that is, the reference point):

[0037] ,

[0038] where, is the median of the data on the reference radius, is the data volume on the reference radius, The th value after arranging all the data on the reference radius in order.

[0039] Step 1.2.3: Compare the division boundary with half of the radius of the reference circle. If the two are infinitely close, stop the grid division; otherwise, recursively repeat Step 1.2.1 - Step 1.2.2 for the divided grid until the division boundary is infinitely close to half of the radius of the reference circle, complete the dynamic grid division, and achieve data clustering with the distance from the reference point as the characteristic quantity; Take as the radius to draw a circle, thus dividing the reference circle into two parts and , and are concentric inner and outer circles, where is the inner circle, is the annular part obtained by removing from the reference circle. At the same time, compare with half of the radius of the reference circle . If the amount of data between and is less than one ten-thousandth of the amount of data on the reference radius , it is determined that the two are infinitely close and stop the grid division; otherwise, recursively perform grid division on and by repeating Step 1.2 until is infinitely close to , then complete the dynamic division of the grid, and the data of each grid realizes clustering with the distance from the reference point as the characteristic quantity.

[0040] Step 1.3: Convert and assemble the data of each grid respectively to generate a data set that conforms to the browser request and has a grid index number.

[0041] Step 1.3.1: Package the data queried from the database according to the structure of the interface agreement and convert it into data in json format; The data format conversion and assembly is to make a one-to-one correspondence between the attributes and attribute values of the data queried from the database according to the structure of the interface agreement, so as to package it into the structure of the interface agreement, and deserialize the java object into json format data through the Jackson library.

[0042] Step 1.3.2: Generate a grid index number according to the division boundary during each grid division to complete the data format conversion and assembly; The grid index number is or , that is, in coordinate form, where is the grid closest to the reference point ( ). Without loss of generality, since is the center of the reference circle. Here, it is also used as the grid index number closest to the center of the reference circle. For differential representation, it is defined that , and the range of data points included in this grid is:

[0043] ,

[0044] is the grid formed farther from the reference point during the grid division process, and the range of data points it includes is:

[0045] .

[0046] Step 1.4: Perform data compression on the data set with grid index numbers to compress the data volume and reduce the transmission delay caused by a large amount of data during data transmission.

[0047] Specifically, use the LZ4 data compression algorithm to compress the data set with grid index numbers to reduce the data volume and prepare for responding to browser requests.

[0048] Step 2: Transmit the compressed data to the browser side through the network, and the browser side decompresses the received data and restores it to the data set with grid index numbers; specifically:

[0049] Step 2.1: Transmit the compressed data as response data in the network to respond to browser requests. Specifically, encapsulate the compressed data as the response body into an HTTP response message including a status line, response headers, a blank line, and the response body, and use the http protocol for data transmission and send it to the browser side.

[0050] Step 2.2: The browser side receives the response data and performs data decompression, and processes it into the format of the data set with grid index numbers before compression on the server side. Specifically, use the method of reverse processing of the compression on the server side (LZ4 compression algorithm) for the received compressed data to restore the data to the format of the data set with grid index numbers before compression.

[0051] Step 3: Perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight to further reduce the amount of data to be rendered and ensure that the retained data can reflect the characteristics and trends of all data.

[0052] Step 3.1: Calculate the density weight of each grid based on the data range and data volume within the grid, and determine the sampling data volume of each grid according to the density weight; density analysis is the process of calculating the data density weight of each grid. The density weight of the grid with index number is The calculation method is as follows:

[0053]

[0054] Among them, is the density weight of the grid with the index number ; is the index number of the adjacent grid; is the data volume of the grid with the index number ; is the total data volume.

[0055] Calculate the sampled data volume of each grid based on the data density weight :

[0056] ,

[0057] Step 3.2: Randomly select sampled data within the grid range according to the sampled data volume of each grid. So far, all the data to be rendered has been prepared in the way of grid division.

[0058] Step 4: Map the reference circle divided by the grid according to the rendering visible area in the browser, and perform hierarchical rendering of the mapped grid in the browser based on the grid density weight. When rendering for the first time, give priority to rendering the grid data with a higher density weight, and render the remaining grid data when the user zooms in on the view.

[0059] Step 4.1: Obtain the range of the rendering visible area in the browser, and perform translation and mapping on the reference circle based on the rendering visible area to ensure that the coordinate systems of the reference circle and the rendering visible area are unified;

[0060] First, obtain the range of the rendering visible area in the browser, that is, the upper left corner coordinates and the lower right corner coordinates , and perform translation on the reference circle based on the rendering visible area to ensure that the coordinate systems of the reference circle and the rendering visible area are unified. Finally, make:

[0061] ,

[0062] Step 4.2: Based on the grid density weight, divide the grid into the first layer and the second layer. When rendering for the first time, give priority to rendering the grid data of the first layer with a higher density weight, and monitor the user's zoom behavior of the view. When the user zooms in on the view, render the grid data of the second layer;

[0063] Perform mapping of the rendering visible area on the reference circle, making , where is the magnification factor (usually, when the data scale is large, the mapping of the reference circle is to shrink it, so usually ), To take and the smaller value. Since the data visualization is denser when the data scale is large, hierarchical rendering is adopted in the first rendering. Specifically, first perform a quick sort on the of each grid, select one-half of the grids from largest to smallest as the first layer, and the remaining one-half of the grids as the second layer. Only the data of the first layer grids is rendered in the first rendering, and at the same time, monitor the user's zoom behavior on the view. When it is monitored that the user zooms in on the view, render the data of the second layer grids.

[0064] In summary, the optimization process of the present invention covers the entire process of data visualization such as data processing, data transmission, and data rendering, fully excavates the performance consumption points and time delay points existing in each process and improves them, and can greatly improve the visualization speed and performance of massive data and improve the user experience. By introducing the grid division of two-dimensional data and abandoning the traditional rectangular grid according to the characteristics of data visualization and adopting the method of benchmark circle for grid division, it can effectively avoid the omission of boundary data and ensure the integrity of data. At the same time, in the downsampling process, fully consider the data density and spatial range of each grid, and perform downsampling of each grid based on the density weight. While reducing the amount of rendered data, effectively ensure that the retained data can accurately reflect the data distribution characteristics and trends, and improve the accuracy of visualization. Finally, in the rendering process, layer the data based on the grid density weight, and realize the hierarchical rendering of data by monitoring user behavior, accelerate the rendering efficiency, greatly shorten the first rendering time, and realize the fast, real-time, and accurate visualization rendering of massive data.

[0065] On the other hand, the present invention provides a high-performance massive data fast visualization rendering optimization device, and each module included therein can implement each step of the foregoing method. Specifically, it includes:

[0066] A preprocessing module, used to screen, deduplicate, and perform dynamic grid division on the data in the database according to the request of the browser side, generate a clustering data set with grid indexes, and compress the data set;

[0067] A decompression module, used to transmit the compressed data to the browser side through the network, and the browser side decompresses the received data to restore it to a data set with grid indexes;

[0068] A sampling module, used to perform density analysis on the decompressed data set, calculate the density weight of each grid, and perform downsampling on the data of each grid based on the data density weight;

[0069] A rendering module, configured to map the reference circle divided by the grid according to the rendering visible area in the browser, and perform hierarchical rendering of the mapped grid in the browser based on the grid density weight.

[0070] In a third aspect, the present invention provides an electronic device, including: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the foregoing high-performance massive data fast visualization rendering optimization method.

[0071] In a fourth aspect, the present invention provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is enabled to implement the foregoing high-performance massive data fast visualization rendering optimization method.

[0072] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A high-performance mass data fast visualization rendering optimization method, characterized in that: The method comprises: Step 1: According to the request of the browser, the server screens, removes duplicates and dynamically divides the data in the database into grids, generates a clustered data set with a grid index number, and compresses the data set; Step 2: The compressed data is transmitted to the browser via the network, and the browser decompresses the received data and restores it to a data set with a grid index number; Step 3: Perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight; Step 4: Map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight; wherein, The step 1 comprises: Step 1.1: The server performs conditional analysis on the request sent by the browser, extracts conditional information to generate a database query statement, filters the data that meets the conditions in the database through the query statement, performs Bloom filter deduplication on the filtered data, and obtains preliminary response data; Step 1.2: Divide the obtained preliminary response data into multiple grids according to the corresponding characteristic quantities; including: Step 1.2.1, dynamically calculating the center point of the grid division based on the coordinate point information of all data as the reference point, and determining the circular space range centered on the reference point as the reference circle; Step 1.2.2, select a reference radius, calculate the median of the distance between the coordinate point on the reference radius and the center of the reference circle as the division boundary of the grid, and divide the reference circle into two parts with the division boundary as the radius; Step 1.2.3, compare the division boundary with 1 / 2 of the reference circle radius. If the two are infinitely close, stop the grid division. Otherwise, recursively repeat steps 1.2.1-1.2.2 for the divided grid until the division boundary is infinitely close to 1 / 2 of the reference circle radius, complete the dynamic grid division, and realize data clustering with reference point distance as the feature quantity; Step 1.3, convert the data format and assemble the data of each grid respectively, and generate a data set that meets the browser request and has a grid index number; Step 1.4: compress the data set with the grid index number.

2. According to claim 1, a high-performance mass data fast visualization rendering optimization method is characterized in that: The step 1.3 comprises: Step 1.3.1: Encapsulate the data queried from the database according to the structure agreed upon by the interface and convert it into json format data; Step 1.3.2: Generate a grid index number based on the division boundary of each grid division to complete data format conversion and assembly.

3. The high-performance mass data fast visualization rendering optimization method according to claim 1 is characterized in that: The step 2 comprises: Step 2.1, encapsulate the compressed data as a response body into an HTTP response message containing a status line, a response header, a blank line and a response body, and use the http protocol for data transmission to send it to the browser end; Step 2.2: The browser receives the response body, decompresses the data, and processes it into the format of the data set with the grid index number before compression by the server.

4. The high-performance mass data fast visualization rendering optimization method according to claim 1 is characterized in that: The step 3 comprises: Step 3.1, calculate the density weight of each grid based on the data range and data volume in the grid, and determine the sampling data volume of each grid according to the density weight; Step 3.2: Randomly select sampling data within the grid range according to the sampling data volume of each grid to complete the preparation of the data to be rendered.

5. The high-performance mass data fast visualization rendering optimization method according to claim 1 is characterized in that: The step 4 comprises: Step 4.1, obtaining the range of the rendering visible area in the browser, translating and mapping the reference circle based on the rendering visible area, and ensuring that the reference circle and the rendering visible area have the same coordinate system; Step 4.2: Based on the grid density weight, the grid is divided into the first layer and the second layer. When rendering for the first time, the first layer of grid data with a higher density weight is rendered first, and the user's zooming behavior of the view is monitored. When the user zooms in on the view, the second layer of grid data is rendered.

6. A high-performance mass data fast visualization rendering optimization device, characterized in that: include: The preprocessing module is used to filter, remove duplicates and dynamically grid the data in the database according to the request of the browser, generate a cluster data set with a grid index number, and compress the data set; it is specifically performed in the following steps: Step 1.1: The server performs conditional analysis on the request sent by the browser, extracts conditional information to generate a database query statement, filters the data that meets the conditions in the database through the query statement, performs Bloom filter deduplication on the filtered data, and obtains preliminary response data; Step 1.2: Divide the obtained preliminary response data into multiple grids according to the corresponding characteristic quantities; including: Step 1.2.1, dynamically calculating the center point of the grid division based on the coordinate point information of all data as the reference point, and determining the circular space range centered on the reference point as the reference circle; Step 1.2.2, select a reference radius, calculate the median of the distance between the coordinate point on the reference radius and the center of the reference circle as the division boundary of the grid, and divide the reference circle into two parts with the division boundary as the radius; Step 1.2.3, compare the division boundary with 1 / 2 of the reference circle radius. If the two are infinitely close, stop the grid division. Otherwise, recursively repeat steps 1.2.1-1.2.2 for the divided grid until the division boundary is infinitely close to 1 / 2 of the reference circle radius, complete the dynamic grid division, and realize data clustering with reference point distance as the feature quantity; Step 1.3, convert the data format and assemble the data of each grid respectively, and generate a data set that meets the browser request and has a grid index number; Step 1.4, compressing the data set with the grid index number; The decompression module is used to transmit the compressed data to the browser through the network. The browser decompresses the received data and restores it to a data set with a grid index number. The sampling module is used to perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight; The rendering module is used to map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement a high-performance, massive data fast visualization rendering optimization method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by the processor, the processor can implement a high-performance mass data fast visualization rendering optimization method as described in any one of claims 1-5.

Images