A database buffer management system

Abstract

The present invention particularly relates to a database buffer management system. The database buffer management system maps files to virtual memory spaces in units of data files or data file segments, thereby collecting statistical information of database buffers and workloads; predicting the next time a data file or a data file segment will be used; and managing the database buffer based on the prediction results, including but not limited to eliminating data from the data buffer or preloading data. The database buffer management system not only optimizes buffer management at a reasonable workload monitoring and prediction cost, but also effectively improves the effectiveness of database buffer management through workload prediction, reduces synchronous I / O operations caused by database read and write operations, and greatly improves the performance of database read and write.

Description

Technical Field

[0001] The present invention relates to the technical field of database management, and in particular to a database buffer management system. Background Art

[0002] The database buffer is an area in the memory allocated by the database management system to cache database data, including tables, indexes and other database objects. The database buffer plays a vital role in the read and write performance of the database. In the use of the database buffer, synchronization operations should be avoided as much as possible, that is, data is synchronized from the disk to the memory only when the data is needed. Otherwise, the database read and write performance will be greatly reduced. Common database buffer management systems include LRU (least recently used), FIFO (first in first out), etc.

[0003] However, the traditional database management buffer method using data pages or similar concepts cannot be associated with the business due to the small management unit, and the additional overhead of monitoring and prediction is so huge that it has no practical value.

[0004] Based on the above situation, the present invention proposes a database buffer management system. Summary of the invention

[0005] In order to make up for the defects of the prior art, the present invention provides a simple and efficient database buffer management system.

[0006] The present invention is achieved through the following technical solutions:

[0007] A database buffer management system, characterized by comprising a file mapping management module, a workload collection module, a workload prediction module and a data buffer management strategy execution module;

[0008] The file mapping management module is responsible for mapping files to virtual memory space in units of data files or data file segments, thereby realizing a database buffer;

[0009] The workload collection module is responsible for collecting and recording the workload profile information of the database in units of data files or data file segments, collecting workload statistics, and cleaning up the workload profile information of the database that is expired and has been used for analysis;

[0010] The workload prediction module is responsible for preprocessing the information collected by the workload collection module and establishing an artificial intelligence model to predict the time when a data file or a data file segment will be used next;

[0011] The data buffer management strategy execution module is responsible for managing the database buffer based on the prediction result of the workload prediction module, including but not limited to eliminating data from the data buffer or preloading data.

[0012] In order to improve the database reading and writing performance, the file mapping management module pre-maps or pre-loads the data files or data file segments that will be used, that is, maps the data files or data file segments that will be used to the virtual memory space in advance before the data is used;

[0013] When the database performs read and write operations, it first checks whether the objects required for the database read and write operations have been mapped to the virtual memory space; if not, it first maps the required data files or data file fragments to the virtual memory space, and then performs subsequent data read and write operations from the virtual memory space;

[0014] If the data is no longer used, an unmap operation is performed on the data file or the data file segment, that is, the mapping of the data file or the data file segment to the virtual memory space is released.

[0015] In order to meet the mapping needs of data files of various sizes, the file mapping management module supports mapping a data file or a data file segment to multiple discontinuous virtual memory spaces.

[0016] On Linux, Unix and MacOS operating systems, the memory mapping file mechanism provided by the operating system is used to implement mapping and de-mapping from data files or data file fragments to virtual memory space.

[0017] The workload profile information of the database includes but is not limited to the time of workload collection, the usage of data files and other information reflecting the usage of data files;

[0018] The usage of the data file refers to whether the data file is being read, written, or not being operated (no operation means no record);

[0019] Other information reflecting the usage of data files refers to whether they are compressed, called or marked.

[0020] In order to save system resources, the workload profile information of the database is summarized in the form of statistical information, and the summarized workload profile information is not limited to the time period of workload collection, the segment of the data file, the number of read times of the data file, the number of read lines of the data file, the number of write times of the data file, the number of written lines of the data file, the number of update times of the data file, the number of updated lines of the data file and other information reflecting the usage of the data file;

[0021] The other information reflecting the usage of the data file indicates the number of times it is compressed, called or marked.

[0022] For the workload of the time series database, when the workload prediction module performs data preprocessing on the collected information, new features are generated according to the collection time of the workload and the generation time of the data in the data file, or grouping transformation is performed in units of 1 hour or 6 hours according to the collection time to generate new data features.

[0023] The workload prediction module, based on the information collected by the workload collection module, uses a time series prediction algorithm (such as an autoregressive integrated moving average model ARIMA, a long short-term memory network LSTM, a gated recurrent unit GRU or a one-dimensional convolutional neural network ConvNets, etc.) to predict the probability of use of a custom-specified data file at a custom time, the time the data file is expected to be used after the custom time point, and the amount of data that is expected to increase in the data file at the custom time point.

[0024] In the workload of the time series database, the workload prediction module predicts the estimated usage time and data quantity of the data files expected to be generated within a custom time.

[0025] The data buffer management strategy execution module is responsible for determining whether and when to perform reverse mapping on the file mapping in the data buffer, whether and when to perform pre-mapping on the file for which the file mapping has not been established, and determining the file segmentation method of the mapped file;

[0026] When determining whether a data file needs to be de-mapped, the data buffer management strategy execution module prioritizes de-mapping data files with low probability weights based on the data file usage probability weights and actual resource conditions;

[0027] When deciding whether and when a data file is pre-mapped, the data buffer management strategy execution module first adds a data file to the corresponding queue according to the time it is expected to be used; when the corresponding queue needs to be executed, the data file with a high pre-mapping probability weight is given priority based on the data file usage probability weight and actual resource conditions.

[0028] The data file is defined using probability weights W as follows:

[0029]

[0030] Wherein, T is the current time, and T' is the time when the file or partition may be used next time predicted by the time series data workload prediction module, that is, the time when the partition needs to be mapped to the buffer.

[0031] Pre-mapping of data files is to map files that are about to be used into virtual memory space in advance, or to map file segments that are about to be expanded due to insufficient space into virtual memory space in advance.

[0032] When determining the segmentation of a data file, the data buffer management strategy execution module calculates the consistency of the probability weights of each segment of data in the data file; for segments with large consistency differences, a segment mapping method is used to perform independent mapping and de-mapping, or map them to discontinuous virtual memory spaces.

[0033] The beneficial effects of the present invention are as follows: the database buffer management system not only optimizes the buffer management under reasonable workload monitoring and prediction costs, but also effectively improves the effectiveness of database buffer management through workload prediction, reduces the synchronous I / O operations caused by database read and write operations, and greatly improves the database read and write performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Attached Figure 1 It is a schematic diagram of the database buffer management method of the present invention.

[0036] Attached Figure 2 The figure is a flow chart of the buffer management phase in the data insertion scenario of the present invention.

[0037] Attached Figure 3 The figure is a flow chart of the buffer management phase in the data query scenario of the present invention.

[0038] Attached Figure 4 This is a schematic diagram of the buffer management strategy update process of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] Memory mapping file technology is a technology that associates a section of virtual memory with a file or a section of a file. Memory mapping technology can significantly improve I / O access performance. This is mainly due to the following two points:

[0041] First, reading and writing memory-mapped files are operations on virtual memory, which are much faster than I / O system calls provided by the operating system.

[0042] Secondly, since the memory area used by the memory-mapped file is located in the operating system kernel, the reading and writing of the memory-mapped file can be directly operated in the kernel's memory space, avoiding the memory copying of data between the kernel space and the user space.

[0043] However, memory-mapped files do not have the above performance advantages in all scenarios. The read and write performance of memory-mapped files is often poor for small files. In addition, if a large number of page faults are generated when reading and writing memory-mapped files, the read and write performance will be inferior to traditional I / O methods. Furthermore, the mapping and unmapping operations of memory-mapped files are relatively time-consuming, and the system will be unavailable during the file unmapping process.

[0044] In addition, since the reading and writing mechanism of memory-mapped files is different from that of traditional files, the design of the data buffer system based on memory-mapped files is different from that of the data buffer system of traditional databases. It needs to maximize the advantages of memory-mapped file technology and avoid the disadvantages mentioned above.

[0045] A time series database is a data management system optimized for the storage and management of time series data. Compared with the data processed by traditional databases, time series data has the following characteristics, such as:

[0046] 1) Data always contains timestamp attributes and is usually generated frequently and stably with time as the dimension;

[0047] 2) Huge amount of data. For example, a sensor generates about 1.4M data in a day. A factory with tens of millions of sensors can generate 500T of data in a year. In emerging industries such as Internet of Vehicles and digital energy, managing hundreds of millions or even billions of devices is becoming a trend.

[0048] 3) Data is updated and deleted less frequently;

[0049] 4) Queries usually use time as a filtering condition, and aggregate analysis is usually performed on the device and time dimensions.

[0050] The database buffer plays a vital role in the read and write performance of the database. Since the read and write speed of data in memory is 2-3 orders of magnitude faster than the read and write speed of data in disk, a high-performance database needs to ensure that the required data is already in the database buffer when the database processes the read and write request, without having to copy the data to the buffer first. The database buffer is a memory area, so the space it can use is limited. Accurately predicting data usage and managing the database buffer accordingly can significantly improve the database read and write performance.

[0051] The database buffer management system includes a file mapping management module, a workload collection module, a workload prediction module and a data buffer management strategy execution module;

[0052] The file mapping management module is responsible for mapping files to virtual memory space in units of data files or data file segments, thereby realizing a database buffer;

[0053] The workload collection module is responsible for collecting and recording the workload profile information of the database in units of data files or data file segments, collecting workload statistics, and cleaning up the workload profile information of the database that is expired and has been used for analysis;

[0054] The workload prediction module is responsible for preprocessing the information collected by the workload collection module and establishing an artificial intelligence model to predict the time when a data file or a data file segment will be used next;

[0055] The data buffer management strategy execution module is responsible for managing the database buffer based on the prediction result of the workload prediction module, including but not limited to eliminating data from the data buffer or preloading data.

[0056] In order to improve the database reading and writing performance, the file mapping management module pre-maps or pre-loads the data files or data file segments that will be used, that is, maps the data files or data file segments that will be used to the virtual memory space in advance before the data is used;

[0057] When the database performs read and write operations, it first checks whether the objects required for the database read and write operations have been mapped to the virtual memory space; if not, it first maps the required data files or data file fragments to the virtual memory space, and then performs subsequent data read and write operations from the virtual memory space;

[0058] If the data is no longer used, an unmap operation is performed on the data file or the data file segment, that is, the mapping of the data file or the data file segment to the virtual memory space is released.

[0059] In order to meet the mapping needs of data files of various sizes, the file mapping management module supports mapping a data file or a data file segment to multiple discontinuous virtual memory spaces.

[0060] On Linux, Unix and MacOS operating systems, the memory mapping file mechanism provided by the operating system is used to implement mapping and de-mapping from data files or data file fragments to virtual memory space.

[0061] The workload profile information of the database includes but is not limited to the time of workload collection, the usage of data files and other information reflecting the usage of data files;

[0062] The usage of the data file refers to whether the data file is being read, written, or not being operated (no operation means no record);

[0063] Other information reflecting the usage of data files refers to whether they are compressed, called or marked.

[0064] Since complete workload collection will occupy a large amount of system resources and even affect the business workload, the workload profile information of the database is summarized in the form of statistical information. The summarized workload profile information includes but is not limited to the time period of workload collection (such as in units of 5 minutes), the segment of the data file (such as in units of 1,000 lines), the number of times the data file is read, the number of lines read from the data file, the number of times the data file is written, the number of lines written to the data file, the number of times the data file is updated, the number of lines updated in the data file, and other information reflecting the usage of the data file;

[0065] The other information reflecting the usage of the data file indicates the number of times it is compressed, called or marked.

[0066] When the workload prediction module performs data preprocessing on the collected information, it performs feature conversion according to the data characteristics to generate new data features;

[0067] Specifically, for the workload of the time series database, new features are generated according to the collection time of the workload and the generation time of the data in the data file, or grouping transformation is performed in units of 1 hour or 6 hours according to the collection time to generate new data features.

[0068] For example, the time difference between the workload collection time and the data generation time in the data file is used. For example, if the workload collected at '2024-03-08 19:25:00' executes a query to access the data from '2024-03-01' to '2024-03-07' for aggregation, then in the statistical information corresponding to the data file where '2024-03-01' is located, in addition to the collection time '2024-03-08', -7Days (indicating 7 days ago) should also be generated as a new data feature.

[0069] The workload prediction module does not limit the way of generating features.

[0070] The workload prediction module, based on the information collected by the workload collection module, uses a time series prediction algorithm (such as an autoregressive integrated moving average model ARIMA, a long short-term memory network LSTM, a gated recurrent unit GRU or a one-dimensional convolutional neural network ConvNets, etc.) to predict the probability of use of a custom-specified data file at a custom time, the time the data file is expected to be used after the custom time point, and the amount of data that is expected to increase in the data file at the custom time point.

[0071] The AI ​​model also predicts other types of information related to the usage of a data file at a certain point in time.

[0072] The workload prediction module not only predicts the existing data files, but also predicts the estimated usage time and data quantity of the data files expected to be generated within a custom time in the workload of the time series database.

[0073] The above methods are particularly effective for workload prediction of time series data, but they can also produce varying degrees of effect on other workloads.

[0074] The data buffer management strategy execution module is responsible for determining whether and when to perform reverse mapping on the file mapping in the data buffer, whether and when to perform pre-mapping on the file for which the file mapping has not been established, and determining the file segmentation method of the mapped file;

[0075] When determining whether a data file needs to be de-mapped, the data buffer management strategy execution module prioritizes de-mapping data files with low probability weights based on the data file usage probability weights and actual resource conditions;

[0076] When deciding whether and when a data file is pre-mapped, the data buffer management strategy execution module first adds a data file to the corresponding queue according to the time it is expected to be used. When the corresponding queue needs to be executed, it prioritizes pre-mapping of data files with high probability weights based on the data file usage probability weights and actual resource conditions.

[0077] The data file is defined using probability weights W as follows:

[0078]

[0079] Wherein, T is the current time, and T' is the time when the file or partition may be used next time predicted by the time series data workload prediction module, that is, the time when the partition needs to be mapped to the buffer.

[0080] The above definition is only an example. Any operation that sets weights using parameters such as prediction time and current time and selects data files or data file fragment partitions with small weights for reverse mapping may produce beneficial effects and comply with the method proposed in the present invention.

[0081] Pre-mapping of data files is to map files that are about to be used into virtual memory space in advance, or to map file segments that are about to be expanded due to insufficient space into virtual memory space in advance.

[0082] When determining the segmentation of a data file, the data buffer management strategy execution module calculates the consistency of the probability weights of each segment of data in the data file; for segments with large consistency differences, a segment mapping method is used to perform independent mapping and de-mapping, or map them to discontinuous virtual memory space.

[0083] The database buffer management system is applicable to all systems that map files to virtual memory space and perform workload prediction based on data files or data file fragments. In particular, in a time series data management system based on memory mapping file technology, since the data mapping unit (such as the entire file or a large-scale file fragment) in the system based on memory mapping file technology is large, and the workload of the time series database has a good characteristic of changing according to time rules, the database buffer management system has achieved particularly outstanding beneficial effects.

[0084] Table 1 Comparison of different buffer management

[0085]

[0086] As described in Table 1, the use of the database buffer management system can improve the throughput of the selected time series workload by 20% to several times.

[0087] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A database buffer management system, characterized in that: It includes a file mapping management module, a workload collection module, a workload prediction module and a data buffer management strategy execution module; The file mapping management module is responsible for mapping files to virtual memory space in units of data files or data file segments, thereby realizing a database buffer; The workload collection module is responsible for collecting and recording the workload profile information of the database in units of data files or data file fragments, realizing the collection of workload statistics, and clearing the workload profile information of the database that is expired and has been used for analysis; The workload prediction module is responsible for preprocessing the information collected by the workload collection module and establishing an artificial intelligence model to predict the time when a data file or a data file segment will be used next; The workload prediction module uses a time series prediction algorithm to predict the usage probability of a custom-specified data file at a custom time, the time when the data file is expected to be used after the custom time point, and the amount of data that is expected to be added to the data file at the custom time point based on the information collected by the workload collection module; The data buffer management strategy execution module is responsible for managing the database buffer based on the prediction result of the workload prediction module, including but not limited to eliminating data from the data buffer or preloading data.

2. The database buffer management system according to claim 1, characterized in that: In order to improve the database reading and writing performance, the file mapping management module pre-maps or pre-loads the data files or data file segments that will be used, that is, maps the data files or data file segments that will be used to the virtual memory space in advance before the data is used; When the database performs read and write operations, it first checks whether the objects required for the database read and write operations have been mapped to the virtual memory space; If not, the required data file or data file fragment is first mapped to the virtual memory space, and then subsequent data read and write operations are performed from the virtual memory space; If the data is no longer used, a demapping operation is performed on the data file or the data file segment, that is, the mapping of the data file or the data file segment to the virtual memory space is released.

3. The database buffer management system according to claim 2, characterized in that: In order to meet the mapping needs of data files of various sizes, the file mapping management module supports mapping a data file or a data file segment to multiple discontinuous virtual memory spaces; On Linux, Unix and MacOS operating systems, the memory mapping file mechanism provided by the operating system is used to implement mapping and de-mapping from data files or data file fragments to virtual memory space.

4. The database buffer management system according to claim 1, characterized in that: The workload profile information of the database includes but is not limited to the time of workload collection, the usage of data files and other information reflecting the usage of data files; The usage of the data file refers to whether the data file is being read, written or not being operated; Other information reflecting the usage of data files refers to whether they are compressed, called or marked.

5. The database buffer management system according to claim 4, characterized in that: In order to save system resources, the workload profile information of the database is summarized in the form of statistical information, and the summarized workload profile information is not limited to the time period of workload collection, the segment of the data file, the number of read times of the data file, the number of read lines of the data file, the number of write times of the data file, the number of written lines of the data file, the number of update times of the data file, the number of updated lines of the data file and other information reflecting the usage of the data file; The other information reflecting the usage of the data file indicates the number of times it is compressed, called or marked.

6. The database buffer management system according to claim 1, characterized in that: For the workload of the time series database, when the workload prediction module performs data preprocessing on the collected information, new features are generated according to the workload collection time and the data generation time in the data file, or grouping transformation is performed in units of 1 hour or 6 hours according to the collection time to generate new data features; The workload prediction module predicts the estimated usage time and data quantity of the data files that are expected to be generated within the custom time.

7. The database buffer management system according to claim 1, characterized in that: The data buffer management strategy execution module is responsible for determining whether and when to perform reverse mapping on the file mapping in the data buffer, whether and when to perform pre-mapping on the file for which the file mapping has not been established, and determining the file segmentation method of the mapped file; When determining whether a data file needs to be de-mapped, the data buffer management strategy execution module prioritizes de-mapping data files with low probability weights based on the data file usage probability weights and actual resource conditions; When determining whether and when a data file is to be pre-mapped, the data buffer management strategy execution module first adds a data file to a corresponding queue according to the time when the data file is expected to be used; when the corresponding queue needs to be executed, the data file with a high probability weight is pre-mapped first according to the probability weight of the data file use and the actual resource situation; Pre-mapping of data files is to map files that are about to be used into virtual memory space in advance, or to map file segments that are about to be expanded due to insufficient space into virtual memory space in advance.

8. The database buffer management system according to claim 7, characterized in that: The data file is defined using probability weights W as follows: Wherein, T is the current time, and T' is the time when the file or partition may be used next time predicted by the time series data workload prediction module, that is, the time when the partition needs to be mapped to the buffer.

9. The database buffer management system according to claim 7, characterized in that: When determining the segmentation of a data file, the data buffer management strategy execution module calculates the consistency of the probability weights of each segment of data in the data file; for segments with large consistency differences, a segment mapping method is used to perform independent mapping and de-mapping, or map them to discontinuous virtual memory space.

Images