Method and device for quickly and visually rendering and optimizing high-performance mass data

Abstract

The invention discloses a high-performance mass data rapid visual rendering optimization method and device, and belongs to the technical field of data processing and visualization. The method comprises the following steps: generating a clustering data set with a grid index number by a server side according to a request of a browser side, and compressing the data set; the compressed data is transmitted to a browser end through a network, and the browser end decompresses the received data and restores the data into a data set with a grid index number; performing density analysis on the decompressed data set, calculating the density weight of each grid, and performing downsampling on the data of each grid; and mapping the reference circle divided by the grid according to the rendering visible area in the browser, and performing layered rendering on the mapped grid in the browser based on the grid density weight. According to the method, the rendering efficiency is improved, the first rendering time is shortened, and rapid, real-time and accurate visual rendering of mass data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and visualization, and specifically relates to a high-performance mass data fast visualization rendering optimization method and device. Background Art

[0002] With the continuous development and popularization of information technology, the application of big data has penetrated into all walks of life and has become an important force in promoting social progress and economic development. However, the visualization and rendering of massive data has always been a problem that has plagued the development of the industry. The traditional data visualization method is to pre-process the data on the server side, process it into the data format defined by the interface, and respond to the browser side for visualization rendering. This method can achieve fast data rendering when the data scale is small, but when the data scale increases, the rendering speed will slow down significantly.

[0003] Many scholars have conducted research on this issue and optimized traditional data processing and rendering methods. These optimizations mainly include adding data screening and deduplication steps during data preprocessing on the server side, and achieving rendering optimization through interpolation and other technologies on the browser side. The optimized methods can improve the processing and optimization of a certain data scale.

[0004] Based on the current research of scholars, the processing and optimization of data scale can be improved to a certain extent. However, when the data scale continues to increase, the overall rendering speed is often slow and the effect is poor due to slow preprocessing speed, large amount of data transmission, and insufficient memory or congestion of the browser when rendering massive data. There are even problems such as freeze, delay and crash of the browser, which cannot meet the user's requirements for real-time and accuracy, and greatly reduces the user experience. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a high-performance optimization method and device for fast visualization rendering of massive data, which optimizes the visualization rendering process of massive data from three aspects: data processing on the server side, data transmission, and data rendering on the browser side, and ultimately solves the problems of slow visualization rendering speed and poor effect of large-scale data, and realizes fast, real-time and accurate visualization rendering of massive data on the browser side.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A high-performance mass data fast visualization rendering optimization method, the method comprising:

[0008] Step 1: According to the request of the browser, the server screens, removes duplicates and dynamically divides the data in the database into grids, generates a clustered data set with a grid index number, and compresses the data set;

[0009] Step 2: The compressed data is transmitted to the browser via the network, and the browser decompresses the received data and restores it to a data set with a grid index number;

[0010] Step 3: Perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight;

[0011] Step 4: Map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight.

[0012] In another aspect, the present invention provides a high-performance mass data fast visualization rendering optimization device, comprising:

[0013] The preprocessing module is used to filter, remove duplicates and dynamically grid the data in the database according to the request of the browser, generate a cluster data set with a grid index number, and compress the data set;

[0014] The decompression module is used to transmit the compressed data to the browser through the network. The browser decompresses the received data and restores it to a data set with a grid index number.

[0015] The sampling module is used to perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight;

[0016] The rendering module is used to map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight.

[0017] In a third aspect, the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned high-performance massive data fast visualization rendering optimization method.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned high-performance, massive data rapid visualization rendering optimization method.

[0019] The beneficial effects of the present invention are:

[0020] Introducing grid division of two-dimensional data and performing grid division based on the reference circle can effectively avoid the omission of boundary data and ensure the integrity of the data.

[0021] Each grid is downsampled based on the density weight, which reduces the amount of rendered data while ensuring that the retained data can accurately reflect the data distribution characteristics and trends, thereby improving the accuracy of visualization.

[0022] The data is layered based on the grid density weight, and the layered rendering of the data is achieved by monitoring user behavior to improve rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of a high-performance, massive data rapid visualization rendering optimization method of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] The present invention provides a high-performance optimization method for fast visualization rendering of massive data, introduces the concept of grid data commonly used in geographic information systems, dynamically divides large-scale data into grids on the server side, performs data format conversion, data assembly and other operations based on the divided grids to obtain response data that can respond to browser requests, and compresses the response data before transmitting, which can reduce the transmission time overhead caused by excessively large data files, receives the compressed response data on the browser side and decompresses it, and performs downsampling based on data density weights, which can reduce the amount of data while ensuring that the rendered data is valid and accurate, and finally adopts layered rendering in the rendering process, performs layered rendering in the browser according to the rendering visible area and grid index number in the browser, speeds up rendering efficiency, and finally realizes fast visualization rendering of large-scale data on the browser side.

[0026] like Figure 1 As shown in FIG. 1 , a flowchart of a high-performance mass data fast visualization rendering optimization method of the present invention is provided. It is assumed that the amount of data used for visualization rendering is in the millions and is a discontinuous discrete value. The network status is good during data transmission, and network delay is ignored. The CPU is used to perform data volume color calculation or fusion processing during rendering. The method specifically includes:

[0027] Step 1: According to the request of the browser, the server screens, removes duplicates and dynamically grids the data in the database, generates a cluster data set with a grid index number, and compresses the data set; specifically:

[0028] Step 1.1: The server performs conditional analysis on the request sent by the browser, extracts conditional information to generate a database query statement, and uses the query statement to filter data that meets the conditions in the database. At the same time, the filtered data is deduplicated using a Bloom filter to eliminate dirty data generated by unreliable data sources, and obtain preliminary response data.

[0029] Step 1.2: Perform dynamic grid division on the obtained preliminary response data according to certain rules, divide all data into multiple grids according to corresponding feature quantities, and realize data clustering. Specifically:

[0030] Step 1.2.1, dynamically calculate the center point of the grid division based on the coordinate point information of all data as the reference point, and determine the circular space range centered on the reference point as the reference circle; since the initial response data obtained needs to be displayed on the interface, there is two-dimensional coordinate point information. The rule for dynamic grid division is to dynamically obtain the center point of the grid division based on the coordinate point information of all data as the reference point:

[0031] ,

[0032] in, are the X-axis and Y-axis coordinates of the reference point, is the maximum X-axis coordinate in the preliminary response data, is the minimum X-axis coordinate in the preliminary response data, is the maximum Y-axis coordinate in the preliminary response data, is the minimum Y-axis coordinate in the preliminary response data.

[0033] The circular spatial range of grid division is determined based on the reference point (hereinafter referred to as the reference circle):

[0034] ,

[0035] in, is the radius of the base circle.

[0036] Step 1.2.2, select a reference radius, calculate the median of the distance between the coordinate point on the reference radius and the center of the reference circle as the grid division boundary, and divide the reference circle into two parts with the division boundary as the radius; select The radius is used as the reference radius, and the median of the distance between the coordinate point on the reference radius and the center of the reference circle (that is, the reference point) is calculated:

[0037] ,

[0038] in, is the median of the data on the reference radius, is the amount of data on the reference radius, is the first row after all data on the reference radius are arranged in order. A numerical value.

[0039] Step 1.2.3, compare the division boundary with 1 / 2 of the reference circle radius. If the two are infinitely close, stop the grid division. Otherwise, recursively repeat steps 1.2.1-1.2.2 for the divided grid until the division boundary is infinitely close to 1 / 2 of the reference circle radius, complete the dynamic grid division, and realize data clustering with the reference point distance as the feature quantity; Draw a circle with the radius as the radius, thereby dividing the base circle into two parts and , and are inner and outer circles with the same center, where For the inner circle, Remove for base circle The annular part outside Radius of reference circle 1 / 2 of the comparison, if and The amount of data between is less than the amount of data on the reference radius If the number is less than one ten-thousandth of and Repeat step 1.2 to perform recursive grid division until and When the distance is infinitely close, the dynamic division of the grid is completed, and the data of each grid is clustered with the distance to the reference point as the characteristic quantity.

[0040] Step 1.3: Convert and assemble the data format of each grid to generate a data set that meets the browser request and has a grid index number.

[0041] Step 1.3.1, encapsulate the data queried from the database according to the structure agreed upon by the interface, and convert it into data in json format; data format conversion and data assembly are to correspond the attributes and attribute values ​​of the data queried from the database one by one according to the structure agreed upon by the interface, thereby encapsulating it into the structure agreed upon by the interface, and deserialize the java object into data in json format through the Jackson library.

[0042] Step 1.3.2: Generate grid index number according to the division boundary of each grid division, and complete data format conversion and assembly; grid index number is or , that is, in coordinate form, where is the grid closest to the reference point ( ), without loss of generality, since is the center of the reference circle. Here, it is also used as the grid index number closest to the center of the reference circle. , the data points contained in the grid range are:

[0043] ,

[0044] The grid formed in the grid division process is far from the reference point, and the data points it contains are as follows:

[0045] .

[0046] Step 1.4: compress the data set with the grid index number to reduce the transmission delay caused by the large amount of data during data transmission.

[0047] Specifically, the LZ4 data compression algorithm is used to compress the data set with the grid index number to reduce the data volume and prepare for responding to the browser request.

[0048] Step 2: The compressed data is transmitted to the browser via the network. The browser decompresses the received data and restores it to a data set with a grid index number. Specifically:

[0049] Step 2.1, the compressed data is transmitted as response data in the network to respond to the browser request. Specifically, the compressed data is encapsulated as a response body into an HTTP response message including a status line, a response header, a blank line and a response body, and the data is transmitted to the browser end using the http protocol.

[0050] Step 2.2: The browser receives the response data and decompresses it, processing it into the format of the data set with the grid index number before the server-side compression. Specifically, the received compressed data is restored to the format of the data set with the grid index number before the compression by using the method of reverse processing with the server-side compression (LZ4 compression algorithm).

[0051] Step 3: Perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight to further reduce the amount of data to be rendered and ensure that the retained data can reflect the characteristics and trends of all data.

[0052] Step 3.1: Calculate the density weight of each grid based on the data range and data volume in the grid, and determine the sampling data volume of each grid according to the density weight; density analysis is the process of calculating the density weight of each grid data. The density weight of the grid The calculation is as follows:

[0053]

[0054] in, The index number is The density weight of the grid; is the index number of the adjacent grid; The index number is The amount of grid data; The total amount of data.

[0055] Calculate the amount of data sampled in each grid based on data density weight :

[0056] ,

[0057] Step 3.2: Randomly select sampling data within the grid range according to the sampling data volume of each grid. At this point, the data to be rendered has been fully prepared according to the grid division method.

[0058] Step 4: Map the grid-divided reference circle according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight. Grid data with higher density weight is rendered first during the first rendering, and the remaining grid data is rendered when the user zooms in on the view.

[0059] Step 4.1, obtaining the range of the rendering visible area in the browser, translating and mapping the reference circle based on the rendering visible area, and ensuring that the reference circle and the rendering visible area have the same coordinate system;

[0060] First, get the rendering visible area range in the browser, that is, the upper left corner coordinates And the lower right corner coordinates , based on the rendering visible area, the reference circle is translated to ensure that the reference circle and the rendering visible area coordinate system are consistent, and finally:

[0061] ,

[0062] Step 4.2: Based on the grid density weight, the grid is divided into the first layer and the second layer. When rendering for the first time, the grid data of the first layer with a higher density weight is rendered first, and the user's zooming behavior of the view is monitored. When the user zooms in on the view, the grid data of the second layer is rendered;

[0063] The base circle is mapped to the visible area so that ,in is the magnification factor (usually, when the data size is large, the mapping of the reference circle is to reduce it, so usually ), To obtain and The smaller value in . Since the data is denser after visualization when the data scale is large, layered rendering is used in the first rendering. Specifically, firstly, Perform quick sorting, select half of the grids from large to small as the first layer, and the remaining half of the grids as the second layer. Only the first layer of grid data is rendered for the first rendering. At the same time, monitor the user's zooming behavior on the view, and render the second layer of grid data when the user zooms in on the view.

[0064] In summary, the optimization process of the present invention covers the entire process of data visualization, such as data processing, data transmission, and data rendering, and fully explores the performance consumption points and time delay points in each process and improves them, which can greatly improve the visualization speed and performance of massive data and improve the user experience. By introducing the grid division of two-dimensional data, the traditional rectangular grid is abandoned according to the characteristics of data visualization and the grid division is performed in the form of a reference circle, which can effectively avoid the omission of boundary data and ensure the integrity of the data. At the same time, the data density and spatial range of each grid are fully considered in the downsampling process, and the downsampling of each grid is performed based on the density weight. While reducing the amount of rendered data, it is effectively ensured that the retained data can accurately reflect the data distribution characteristics and trends, and improve the accuracy of visualization. Finally, the data is layered based on the grid density weight during the rendering process, and the layered rendering of the data is realized by monitoring the user behavior, which speeds up the rendering efficiency, greatly shortens the first rendering time, and realizes the fast, real-time and accurate visualization rendering of massive data.

[0065] On the other hand, the present invention provides a high-performance mass data fast visualization rendering optimization device, which includes various modules that can implement various steps of the above method, specifically including:

[0066] The preprocessing module is used to filter, remove duplicates and dynamically grid the data in the database according to the request of the browser, generate a cluster data set with a grid index number, and compress the data set;

[0067] The decompression module is used to transmit the compressed data to the browser through the network. The browser decompresses the received data and restores it to a data set with a grid index number.

[0068] The sampling module is used to perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight;

[0069] The rendering module is used to map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight.

[0070] In a third aspect, the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned high-performance massive data fast visualization rendering optimization method.

[0071] In a fourth aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned high-performance, massive data rapid visualization rendering optimization method.

[0072] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-performance mass data fast visualization rendering optimization method, characterized in that: The method comprises: Step 1: According to the request of the browser, the server screens, removes duplicates and dynamically divides the data in the database into grids, generates a clustered data set with a grid index number, and compresses the data set; Step 2: The compressed data is transmitted to the browser via the network, and the browser decompresses the received data and restores it to a data set with a grid index number; Step 3: Perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight; Step 4: Map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight.

2. According to claim 1, a high-performance mass data fast visualization rendering optimization method is characterized in that: The step 1 comprises: Step 1.1: The server performs conditional analysis on the request sent by the browser, extracts conditional information to generate a database query statement, filters the data that meets the conditions in the database through the query statement, performs Bloom filter deduplication on the filtered data, and obtains preliminary response data; Step 1.2, dividing the obtained preliminary response data into multiple grids according to corresponding characteristic quantities; Step 1.3, convert the data format and assemble the data of each grid respectively, and generate a data set that meets the browser request and has a grid index number; Step 1.4: compress the data set with the grid index number.

3. A high-performance mass data fast visualization rendering optimization method according to claim 2, characterized in that: The step 1.2 comprises: Step 1.2.1, dynamically calculating the center point of the grid division based on the coordinate point information of all data as the reference point, and determining the circular space range centered on the reference point as the reference circle; Step 1.2.2, select a reference radius, calculate the median of the distance between the coordinate point on the reference radius and the center of the reference circle as the division boundary of the grid, and divide the reference circle into two parts with the division boundary as the radius; Step 1.2.3, compare the dividing boundary with 1 / 2 of the reference circle radius. If the two are infinitely close, stop grid division. Otherwise, recursively repeat steps 1.2.1-1.2.2 for the divided grid until the dividing boundary is infinitely close to 1 / 2 of the reference circle radius, complete the dynamic grid division, and realize data clustering with reference point distance as the feature quantity.

4. The high-performance mass data fast visualization rendering optimization method according to claim 2 is characterized in that: The step 1.3 comprises: Step 1.3.1: Encapsulate the data queried from the database according to the structure agreed upon by the interface and convert it into json format data; Step 1.3.2: Generate a grid index number based on the division boundary of each grid division to complete data format conversion and assembly.

5. The high-performance mass data fast visualization rendering optimization method according to claim 1 is characterized in that: The step 2 comprises: Step 2.1, encapsulate the compressed data as a response body into an HTTP response message containing a status line, a response header, a blank line and a response body, and use the http protocol for data transmission to send it to the browser end; Step 2.2: The browser receives the response body, decompresses the data, and processes it into the format of the data set with the grid index number before compression by the server.

6. The high-performance mass data fast visualization rendering optimization method according to claim 1 is characterized in that: The step 3 comprises: Step 3.1, calculate the density weight of each grid based on the data range and data volume in the grid, and determine the sampling data volume of each grid according to the density weight; Step 3.2: Randomly select sampling data within the grid range according to the sampling data volume of each grid to complete the preparation of the data to be rendered.

7. The high-performance mass data fast visualization rendering optimization method according to claim 1 is characterized in that: The step 4 comprises: Step 4.1, obtaining the range of the rendering visible area in the browser, translating and mapping the reference circle based on the rendering visible area, and ensuring that the reference circle and the rendering visible area have the same coordinate system; Step 4.2: Based on the grid density weight, the grid is divided into the first layer and the second layer. When rendering for the first time, the first layer of grid data with a higher density weight is rendered first, and the user's zooming behavior of the view is monitored. When the user zooms in on the view, the second layer of grid data is rendered.

8. A high-performance mass data fast visualization rendering optimization device, characterized in that: include: The preprocessing module is used to filter, remove duplicates and dynamically grid the data in the database according to the request of the browser, generate a cluster data set with a grid index number, and compress the data set; The decompression module is used to transmit the compressed data to the browser through the network. The browser decompresses the received data and restores it to a data set with a grid index number. The sampling module is used to perform density analysis on the decompressed data set, calculate the density weight of each grid, and downsample the data of each grid based on the data density weight; The rendering module is used to map the reference circle divided by the grid according to the rendering visible area in the browser, and render the mapped grid in layers in the browser based on the grid density weight.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement a high-performance, massive data fast visualization rendering optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by the processor, the processor can implement a high-performance mass data fast visualization rendering optimization method as described in any one of claims 1-7.

Images