Dynamic storage optimization method for structured massive real-time data
A real-time data and dynamic storage technology, which is applied in the field of computer information, can solve the problems of data storage location and space size allocation that cannot be dynamically allocated, work order buffer is incompetent, and memory space utilization rate is reduced, so as to improve the efficiency of data retrieval , meet the real-time requirements, and improve the effect of space utilization
Active Publication Date: 2018-08-24
CHONGQING UNIV
View PDF18 Cites 5 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0003] In occasions that require high real-time data, it is not only necessary to receive and send massive data in real time, but also to store the data in the memory database in real time, and to store the data exceeding the time threshold in the disk history database. Such a complex worksheet The buffer is difficult to do, and maintaining the real-time performance of high-speed data storage is a tricky problem
[0004] Later, after the concept of memory database was proposed, various memory-based real-time databases and products for caching real-time data appeared. Memory-based real-time databases need to cache frequently used or immediately used data in memory, which can be accessed during access. Retrieving data directly from the memory can reduce access to external storage such as disks, thereby improving data access efficiency, but the data storage location and space allocation cannot be dynamically allocated, which reduces the space utilization of the memory; Generally, only one memory index and one database index are used when building an index. When the memory index is merged with the historical database index, no new index can be added, which causes a certain delay in the real-time storage of data.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment
[0055] Example: such as figure 1 and figure 2 Shown; a dynamic storage optimization method for structured massive real-time data, which includes:
[0056] S1: Set data screening norms and time thresholds;
[0057] S2: read structured real-time data;
[0058] S3: Screen and mark the real-time data corresponding to the screening specification;
[0059] S4: Establish a dynamic double buffer area for receiving filtered and marked real-time data;
[0060] Described step S4 also includes:
[0061] S41: Count the ratio of the current marked data volume to the total data volume;
[0062] S42: A first buffer area and a second buffer area with the same capacity are respectively set in the memory;
[0063] S43: Dynamically divide the size of the buffer space occupied by the marked data and the non-marked data in the two buffers according to the statistical ratio;
[0064] S44: After filtering and marking the collected data, write it into the first buffer area according to the mar...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention discloses a dynamic storage optimization method for structured massive real-time data. The method comprises: setting a data discrimination specification and a time threshold; reading data; screening and marking the data; establishing dynamic double buffer areas; establishing a hybrid index for the data in the dynamic buffer area; storing the data in the dynamic buffer area that is less than the time threshold into a dynamic real-time database; if the data in the dynamic buffer area exceeds the time threshold, storing the data exceeding the time threshold into a relationalhistorical database; combining the memory index that exceeds the time threshold in the real-time database with the hard disk index; and after the indices are combined, storing the data in the real-time database into the relational history database. The method disclosed by the present invention has the beneficial effects that: when receiving data, the double buffer areas can be used to receive real-time data alternately, or after processing the real-time data, the real-time data can be received uninterruptedly, so that the memory efficiency is maximized and the delay of receiving data is reduced.
Description
technical field [0001] The invention relates to the field of computer information technology, in particular to a dynamic storage optimization method for structured mass real-time data. Background technique [0002] With the development of science and technology, an important application of big data in the field of cloud computing is the storage and retrieval of massive structured data. [0003] In occasions that require high real-time data, it is not only necessary to receive and send massive data in real time, but also to store the data in the memory database in real time, and to store the data exceeding the time threshold in the disk history database. Such a complex worksheet The buffer is difficult to do, and maintaining the real-time performance of high-speed data storage is a tricky problem. [0004] Later, after the concept of memory database was proposed, various memory-based real-time databases and products for caching real-time data appeared. Memory-based real-time...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F17/30
CPCG06F16/221G06F16/2272G06F16/24539G06F16/284
Inventor张可柴毅黄若山朱燕朱博
OwnerCHONGQING UNIV