Grid-based data visualization method and system based on virtual memory and cache technology

Abstract

The application provides a grid data visualization method and system based on virtual memory and cache technology, and the method comprises the following steps: obtaining target grid data; if the size of the target grid data is greater than a preset storage space, writing the target grid data into virtual memory; writing the target grid data in the virtual memory into a cache; reading the target grid data from the cache by using a parallel algorithm; and converting the target grid data into colors, thereby completing visualization. The embodiment of the application solves the problem that super-large grid data occupies a large amount of memory, and realizes fast visualization expression of grid data.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a gridded data visualization method and system based on virtual memory and caching technology. Background Technology

[0002] With the improvement of geophysical measurement accuracy and the expansion of mapping scope, the size of geophysical potential field gridded data has increased exponentially, posing challenges to data visualization. Gridded data refers to the process of grouping spatially unevenly distributed data into grids of a specific geometric shape, calculating the average value of the data in each grid, and placing it at the center of the grid. This process transforms spatially discrete data into regularly distributed gridded data, suppressing local noise and filling in corresponding blank areas. Simultaneously, it provides a consistent spatial structure for the integration and comparison of different variables, thus reflecting the spatial patterns of objective variables more completely and fully.

[0003] Gridded data is characterized by its large information storage capacity, high precision, and small storage space. Geophysical potential field data, being typical spatially continuous data, perfectly meets the requirements for gridded data storage. Therefore, gridded data, especially regular rectangular grids, has become the main data format for current geophysical models.

[0004] In recent years, with the improvement of measurement accuracy and the large-scale mapping, especially the nationwide and global mapping work, the gridded data being processed is often quite large, exceeding 16384 (rows) * 16384 (columns), and the file data is generally over 1 GB. In the process of visualizing gridded data, directly reading the gridded data into memory at once greatly impacts operating system performance, easily leading to software system crashes and affecting other operations.

[0005] Therefore, it is necessary to study gridded data reading technology for ultra-large, high-precision gridded data to ensure its display efficiency and effect. Summary of the Invention

[0006] This invention provides a gridded data visualization method and system based on virtual memory and caching technology. Its main purpose is to combine virtual memory, caching technology, parallel algorithms and other technologies to achieve rapid visualization of ultra-large gridded data.

[0007] In a first aspect, embodiments of the present invention provide a gridded data visualization method based on virtual memory and caching technology, comprising:

[0008] Acquire target gridded data; if the size of the target gridded data is greater than the preset storage space, write the target gridded data into virtual memory.

[0009] Write the target gridded data in the virtual memory into the cache;

[0010] The target gridded data is read from the cache using a parallel algorithm;

[0011] The target gridded data is converted into colors to complete the visualization.

[0012] Furthermore, it also includes:

[0013] If the size of the target gridded data is smaller than the preset storage space, the target gridded data is directly read out using a parallel algorithm;

[0014] The target gridded data is converted into colors to complete the visualization.

[0015] Furthermore, the step of converting the target gridded data into color to complete the visualization includes:

[0016] Determine whether interpolation of the target gridded data is necessary;

[0017] If necessary, the target gridded data is interpolated using a preset interpolation method to obtain the interpolated target gridded data. The interpolated target gridded data is then converted into color to complete the visualization.

[0018] Furthermore, the step of converting the target gridded data into color to complete the visualization also includes:

[0019] If not needed, the target gridded data can be directly converted into color to complete the visualization.

[0020] Furthermore, the preset interpolation method includes bilinear interpolation and cubic convolution interpolation.

[0021] Furthermore, when the visualization speed is less than a preset speed threshold, the preset interpolation method is the bilinear interpolation;

[0022] When the visualization accuracy is greater than the preset accuracy threshold, the preset interpolation method is the cubic convolution interpolation.

[0023] Furthermore, the preset storage space is 32M.

[0024] Secondly, embodiments of the present invention provide a gridded data visualization system based on virtual memory and caching technology, comprising:

[0025] A virtual module is used to acquire target gridded data. If the size of the target gridded data is greater than the preset storage space, the target gridded data is written into virtual memory.

[0026] The caching module is used to write the target gridded data in the virtual memory into the cache;

[0027] A parallel module is used to read the target gridded data from the cache using a parallel algorithm;

[0028] The visualization module is used to convert the target gridded data into colors, thereby completing the visualization.

[0029] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described gridded data visualization method based on virtual memory and caching technology.

[0030] Fourthly, embodiments of the present invention provide a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described gridded data visualization method based on virtual memory and caching technology.

[0031] This invention proposes a gridded data visualization method and system based on virtual memory and caching technology. First, target gridded data is acquired. If the size of the target gridded data exceeds a preset storage space, it is first written to virtual memory. Then, the target gridded data in virtual memory is written to a cache. A parallel algorithm is used to read the target gridded data from the cache. Finally, the target gridded data is converted to color to complete the visualization. This invention solves the problem of large memory consumption for ultra-large gridded data and achieves rapid visualization of gridded data. Attached Figure Description

[0032] Figure 1 A flowchart illustrating a gridded data visualization method based on virtual memory and caching technology, provided as an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the visualization of target gridded data in an embodiment of the present invention;

[0034] Figure 3 This is an overall flowchart of a gridded data visualization method based on virtual memory and caching technology provided in an embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of bilinear interpolation in an embodiment of the present invention;

[0036] Figure 5 This is a simplified schematic diagram of grid node interpolation in an embodiment of the present invention;

[0037] Figure 6This is a schematic diagram of the original file of test data in an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of the test data after gridding in an embodiment of the present invention;

[0039] Figure 8 This is a visual illustration of the gridded test data in an embodiment of the present invention;

[0040] Figure 9 This is a magnified visualization of the gridded test data in an embodiment of the present invention;

[0041] Figure 10 This is a schematic diagram of the structure of a gridded data visualization system based on virtual memory and caching technology, provided as an embodiment of the present invention.

[0042] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0044] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0045] In the embodiments of this application, "at least one" refers to one or more; "multiple" refers to two or more. In the description of this application, terms such as "first," "second," and "third" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0046] References such as “one embodiment” or “some embodiments” as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the terms “comprising,” “including,” “having,” and variations thereof, as used in this specification, mean “including, but not limited to,” unless otherwise specifically emphasized.

[0047] Figure 1 A flowchart illustrating a gridded data visualization method based on virtual memory and caching technology provided in this embodiment of the invention is shown below. Figure 1 As shown, the method includes:

[0048] S110, acquire target gridded data; if the size of the target gridded data is greater than the preset storage space, write the target gridded data into virtual memory.

[0049] S120, Write the target gridded data in the virtual memory into the cache;

[0050] S130, The target gridded data is read from the cache using a parallel algorithm;

[0051] S140, the target gridded data is converted into color to complete the visualization.

[0052] First, the target gridded data is obtained. In the actual implementation process, the data that needs to be visualized is first converted into gridded data. For example, geophysical data is converted into target gridded data. Gridded data is an important component of geophysical data. Gridded maps have wide applications in data processing and interpretation and are the basic maps and important basis for data interpretation.

[0053] During gridded data visualization, the decision to store the gridded data directly in memory or virtual memory depends on its size. If the gridded data exceeds a certain size, such as being larger than a preset storage space, the target gridded data is written to virtual memory; if the gridded data is not larger than the preset storage space, the target gridded data is written directly to memory.

[0054] For example, the preset storage space can be 32M. If the size of the target gridded data is less than 32M, it will be managed directly by memory; otherwise, virtual memory will be used.

[0055] If virtual memory is used, it cannot be directly applied to parallel algorithms, which are key technologies for improving visualization efficiency, with display efficiency being improved by more than 3 times. In order to be compatible with both virtual memory and parallel algorithms, data caching computation is required to temporarily store the target gridded data to improve display efficiency.

[0056] When the target gridded data size exceeds the preset storage space, the target gridded data is first written to virtual memory. Virtual memory is a memory management technique in computer systems. Virtual memory technology alleviates the problem of insufficient physical memory by using hard disk space as an extension of memory. When system memory usage exceeds 80%, the operating system automatically moves some inactive memory data to the hard disk's "paging file" (virtual memory). During the reading of large gridded data, the data is first read into virtual memory, and by using virtual memory as the object of operation, the gridded data is indirectly manipulated.

[0057] Then, the target gridded data in virtual memory is written to the cache. Data caching technology refers to temporarily storing target gridded data in memory or other high-speed storage media. Its main function is to temporarily store computer data to improve read efficiency. To address the issue that virtual memory data cannot be directly applied to parallel algorithms when displaying extremely large gridded data, a temporary data storage method is adopted. Specifically, during data display, data is read into the cache according to the display area size, and then the target gridded data in the cache is used for visualization.

[0058] Figure 2 This is a schematic diagram illustrating the visualization of target gridded data in an embodiment of the present invention, such as... Figure 2 As shown, in grid data display, the range of the display area is first calculated to cover the number of rows and columns of the target gridded data. Then, the target gridded data within this range is read from virtual memory and stored in a cache array, such as... Figure 2 The system reads in 3 rows and 6 columns of data, performs operations on it, and then releases the target gridded data after execution.

[0059] Parallel algorithms improve the speed of processing large amounts of data through multi-core collaborative computation. C++ provides three parallel programming models: threads, OpenMP, and MPI. This embodiment of the invention uses the OpenMP model for parallel algorithm computation. While C# has the Parallel class in the Tasks namespace for parallel development, comparisons show that its efficiency is far lower than that of parallel algorithms in C++. Therefore, the OpenMP parallel algorithm from C++ is used for processing.

[0060] Then, the target gridded data extracted by the parallel algorithm is converted into color to complete the data visualization. There are two ways to visualize gridded data: one is to convert it into vector mode, i.e., contour lines and contour areas, and the other is raster mode, i.e., grid planar map and planar 3D shaded map. Vector mode is a spatial format and is stored in a spatial data manner, while raster mode directly uses gridded data as data objects.

[0061] This invention proposes a gridded data visualization method based on virtual memory and caching technology. First, the target gridded data is acquired. If the size of the target gridded data exceeds a preset storage space, it is first written to virtual memory. Then, the target gridded data in virtual memory is written to a cache. A parallel algorithm is used to read the target gridded data from the cache. Finally, the target gridded data is converted into color to complete the visualization. This invention solves the problem of large memory consumption for ultra-large gridded data and achieves rapid visualization of gridded data.

[0062] In some embodiments, it also includes:

[0063] If the size of the target gridded data is smaller than the preset storage space, the target gridded data is directly read out using a parallel algorithm;

[0064] The target gridded data is converted into colors to complete the visualization.

[0065] Figure 3 A flowchart illustrating the overall process of a gridded data visualization method based on virtual memory and caching technology provided in this embodiment of the invention is shown below. Figure 3 As shown, the method also includes the following: if the target gridded data is smaller than the preset storage space, i.e., the target gridded data is less than 32M, then the target gridded data can be directly read using a parallel algorithm. The read target gridded data is then converted into color to complete the visualization.

[0066] As one implementation method, the step of converting the target gridded data into color to complete the visualization includes:

[0067] Determine whether interpolation of the target gridded data is necessary;

[0068] If necessary, the target gridded data is interpolated using a preset interpolation method to obtain the interpolated target gridded data. The interpolated target gridded data is then converted into color to complete the visualization.

[0069] Furthermore, the step of converting the target gridded data into color to complete the visualization also includes:

[0070] If not needed, the target gridded data can be directly converted into color to complete the visualization.

[0071] The preset interpolation methods include bilinear interpolation and cubic convolution interpolation.

[0072] Wherein, when the visualization speed is less than a preset speed threshold, the preset interpolation method is the bilinear interpolation;

[0073] When the visualization accuracy is greater than the preset accuracy threshold, the preset interpolation method is the cubic convolution interpolation.

[0074] In gridded data visualization, graphics without interpolation algorithms are prone to displaying a mosaic effect. Applying interpolation algorithms significantly enhances the graphic display effect. This invention combines bilinear interpolation and cubic convolution interpolation algorithms. When simple and fast visualization is required, and the visualization accuracy requirement is not high (i.e., the visualization speed is less than a preset speed threshold), linear interpolation is used. Conversely, when high-precision representation is required (i.e., the visualization accuracy is greater than a preset accuracy threshold), cubic convolution algorithms are used to make the graphics more aesthetically pleasing.

[0075] The preset speed threshold and preset accuracy threshold can be determined according to the actual situation.

[0076] Bilinear interpolation, also known as first-order interpolation, is mathematically a generalization of linear interpolation. It's used to interpolate functions of two variables (such as x and y) on a two-dimensional linear grid. Bilinear interpolation first applies linear interpolation in one direction, then in the other. While each step is linear in both the sampled values ​​and locations, the interpolation as a whole is quadratic in terms of the sampled locations. Bilinear interpolation is one of the most fundamental resampling techniques in computer vision and image processing, also known as bilinear filtering or bilinear texture mapping.

[0077] Figure 4 This is a schematic diagram of bilinear interpolation in an embodiment of the present invention, as shown below. Figure 4 Assuming that it is known , , as well as The grid values ​​are respectively , , and The solution point needs to be found. The grid value.

[0078] First of all Linear interpolation is performed in the direction to obtain , :

[0079] (1)

[0080] (2)

[0081] Then Linear interpolation in the direction yields:

[0082] (3)

[0083] Among them, if first in Directional interpolation, and then in Directional interpolation yields the same result as bilinear interpolation performed in the same order.

[0084] Figure 5 This is a simplified schematic diagram of grid node interpolation in an embodiment of the present invention, as shown below. Figure 5 As shown, the output position point is directly calculated and mapped to point P and its four nearest neighbors in the input grid coordinate system. , , , The distance ratio between nodes is used as the weight, and the attribute value (or grayscale value) is assigned to P based on the calculation of the node and the weight.

[0085] (4)

[0086] (5)

[0087] (6)

[0088] Formula (6) is the specific calculation formula for bilinear interpolation. Wherein... To output the coordinates in the grid data The attribute values ​​(or grayscale values) of the nodes. For the coordinates in the input grid data The attribute values ​​(or grayscale values) of the nodes.

[0089] In this embodiment of the invention, the measured data srtmS6.grd was also used for testing, with 12001 rows * 24001 columns. Figure 6 This is a schematic diagram of the original file of test data in an embodiment of the present invention, such as... Figure 6 As shown, the file size is 1.07G, and the rainbow color scheme has 39 levels of equal area distribution. Figure 7 This is a schematic diagram of the test data after gridding in an embodiment of the present invention, as shown below. Figure 7 As shown, the process flow indicates that the entire program's memory usage is 27.040MB, which is much smaller than the file size of 1.07GB. The display effect is as follows... Figure 8 .

[0090] like Figure 9 As shown, Figure 8 Zooming in to the black area took 0.065 seconds, which fully meets the requirements for rapid visualization of ultra-large grids and achieves the design requirements.

[0091] Figure 10A schematic diagram of the structure of a gridded data visualization system based on virtual memory and caching technology provided in an embodiment of the present invention is shown below. Figure 10 As shown, the system includes:

[0092] Virtual module A10 is used to acquire target gridded data. If the size of the target gridded data is greater than the preset storage space, the target gridded data is written into virtual memory.

[0093] Cache module A20 is used to write the target gridded data in the virtual memory into the cache;

[0094] Parallel module A30 is used to read the target gridded data from the cache using a parallel algorithm;

[0095] The visualization module A40 is used to convert the target gridded data into colors, thereby completing the visualization.

[0096] This embodiment is a system embodiment corresponding to the above method. Its specific implementation process is the same as that of the above method embodiment. For details, please refer to the above method embodiment. This system embodiment will not repeat the details.

[0097] The modules in the aforementioned gridded data visualization system based on virtual memory and caching technology can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0098] This invention also provides a computer device, which may be a server. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a computer storage medium and internal memory. The computer storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the computer storage medium. The database stores data generated or acquired during the execution of a gridded data visualization method based on virtual memory and caching technology. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a gridded data visualization method based on virtual memory and caching technology.

[0099] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a gridded data visualization method based on virtual memory and caching technology as described in the above embodiment. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in this embodiment of a gridded data visualization system based on virtual memory and caching technology.

[0100] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the steps of the gridded data visualization method based on virtual memory and caching technology described in the above embodiment. Alternatively, when executed by a processor, the computer program implements the functions of each module / unit in this embodiment of the gridded data visualization system based on virtual memory and caching technology.

[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0103] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A gridded data visualization method based on virtual memory and caching technology, characterized in that, include: Acquire target gridded data; if the size of the target gridded data is greater than the preset storage space, write the target gridded data into virtual memory. Write the target gridded data in the virtual memory into the cache; The target gridded data is read from the cache using a parallel algorithm; The target gridded data is converted into colors to complete the visualization.

2. The gridded data visualization method based on virtual memory and caching technology according to claim 1, characterized in that, Also includes: If the size of the target gridded data is smaller than the preset storage space, the target gridded data is directly read out using a parallel algorithm; The target gridded data is converted into colors to complete the visualization.

3. The gridded data visualization method based on virtual memory and caching technology according to claim 1 or 2, characterized in that, The step of converting the target gridded data into color to complete the visualization includes: Determine whether interpolation of the target gridded data is necessary; If necessary, the target gridded data is interpolated using a preset interpolation method to obtain the interpolated target gridded data. The interpolated target gridded data is then converted into color to complete the visualization.

4. The gridded data visualization method based on virtual memory and caching technology according to claim 3, characterized in that, The target gridded data is converted into color to complete the visualization. The steps also include: If not needed, the target gridded data can be directly converted into color to complete the visualization.

5. The gridded data visualization method based on virtual memory and caching technology according to claim 3, characterized in that, The preset interpolation methods include bilinear interpolation and cubic convolution interpolation.

6. The gridded data visualization method based on virtual memory and caching technology according to claim 5, characterized in that, When the visualization speed is less than a preset speed threshold, the preset interpolation method is the bilinear interpolation. When the visualization accuracy is greater than the preset accuracy threshold, the preset interpolation method is the cubic convolution interpolation.

7. The gridded data visualization method based on virtual memory and caching technology according to any one of claims 1 to 6, characterized in that, The preset storage space is 32M.

8. A gridded data visualization system based on virtual memory and caching technology, characterized in that, include: A virtual module is used to acquire target gridded data. If the size of the target gridded data is greater than the preset storage space, the target gridded data is written into virtual memory. The caching module is used to write the target gridded data in the virtual memory into the cache; A parallel module is used to read the target gridded data from the cache using a parallel algorithm; The visualization module is used to convert the target gridded data into colors, thereby completing the visualization.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the gridded data visualization method based on virtual memory and caching technology as described in any one of claims 1 to 7.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the gridded data visualization method based on virtual memory and caching technology as described in any one of claims 1 to 7.

Images