A dynamic storage optimization method for structured massive real-time data
A real-time data, dynamic storage technology, applied in the field of computer information, can solve the problems of data storage location and space size allocation cannot be dynamically allocated, work order buffer is incompetent, reduce memory space utilization, etc., to improve data retrieval efficiency. , to meet real-time requirements and improve the effect of space utilization
Active Publication Date: 2021-07-30
CHONGQING UNIV
View PDF18 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] In occasions that require high real-time data, it is not only necessary to receive and send massive data in real time, but also to store the data in the memory database in real time, and to store the data exceeding the time threshold in the disk history database. Such a complex worksheet The buffer is difficult to do, and maintaining the real-time performance of high-speed data storage is a tricky problem
[0004] Later, after the concept of memory database was proposed, various memory-based real-time databases and products for caching real-time data appeared. Memory-based real-time databases need to cache frequently used or immediately used data in memory, which can be accessed during access. Retrieving data directly from the memory can reduce access to external storage such as disks, thereby improving data access efficiency, but the data storage location and space allocation cannot be dynamically allocated, which reduces the space utilization of the memory; Generally, only one memory index and one database index are used when building an index. When the memory index is merged with the historical database index, no new index can be added, which causes a certain delay in the real-time storage of data.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment
[0055] Example: such as figure 1 and figure 2 Shown; a dynamic storage optimization method for structured massive real-time data, which includes:
[0056] S1: Set data screening norms and time thresholds;
[0057] S2: read structured real-time data;
[0058] S3: Screen and mark the real-time data corresponding to the screening specification;
[0059] S4: Establish a dynamic double buffer area for receiving filtered and marked real-time data;
[0060] Described step S4 also includes:
[0061] S41: Count the ratio of the current marked data volume to the total data volume;
[0062] S42: A first buffer area and a second buffer area with the same capacity are respectively set in the memory;
[0063] S43: Dynamically divide the size of the buffer space occupied by the marked data and the non-marked data in the two buffers according to the statistical ratio;
[0064] S44: After filtering and marking the collected data, write it into the first buffer area according to the mar...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a dynamic storage optimization method for structured massive real-time data, which includes: setting data screening norms and time thresholds; reading data; screening and marking data; establishing dynamic double buffer areas; Create a hybrid index for the data; store the data in the dynamic cache area that is less than the time threshold into the dynamic real-time database; if the data in the dynamic cache area exceeds the time threshold, store the data exceeding the time threshold in the relational historical database; The memory index exceeding the time threshold in the real-time database is merged with the hard disk index; after the index is merged, the data in the real-time database is stored in the relational historical database. The beneficial effects obtained by the present invention are: when receiving data, double buffers can be used to alternately receive real-time data, or the real-time data can be received uninterruptedly after the real-time data is processed, so that the memory efficiency can be maximized and the delay of receiving data can be reduced sex.
Description
technical field [0001] The invention relates to the field of computer information technology, in particular to a dynamic storage optimization method for structured mass real-time data. Background technique [0002] With the development of science and technology, an important application of big data in the field of cloud computing is the storage and retrieval of massive structured data. [0003] In occasions that require high real-time data, it is not only necessary to receive and send massive data in real time, but also to store the data in the memory database in real time, and to store the data exceeding the time threshold in the disk history database. Such a complex worksheet The buffer is difficult to do, and maintaining the real-time performance of high-speed data storage is a tricky problem. [0004] Later, after the concept of memory database was proposed, various memory-based real-time databases and products for caching real-time data appeared. Memory-based real-time...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & AuthorityPatents(China)
IPC IPC(8): G06F16/22G06F16/28G06F16/2453
CPCG06F16/221G06F16/2272G06F16/24539G06F16/284
Inventor张可柴毅黄若山朱燕朱博
OwnerCHONGQING UNIV