A transaction-based method for reading and writing large-capacity data

A data reading and writing, large-capacity technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of linear growth of data file throughput, high pressure on the file system, poor performance efficiency, etc., to improve read Write access performance, meet business needs, and improve throughput

Active Publication Date: 2016-01-13
上交所技术有限责任公司
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, using the traditional method to write each data record into the data file as a transaction submission, then in the case of a large amount of data and high real-time requirements, frequent IO operations can easily touch the bottleneck of the operating system, resulting in data file corruption. Throughput cannot grow linearly with the frequency of write operations
[0005] If you use the method of packaging data records and submitting them in batches, although IO operations can be reduced, due to the large amount of data, once a transaction rollback occurs, the pressure on the file system will be very high; and for files with a large amount of data, when you need to execute When reading and accessing data records with a specified serial number, if there is no index mechanism, it will take a lot of IO operations to traverse the entire data file, and the performance efficiency will be poor

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A transaction-based method for reading and writing large-capacity data
  • A transaction-based method for reading and writing large-capacity data
  • A transaction-based method for reading and writing large-capacity data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0065] Using the read-write method of the present invention on a single host, its read-write system is as figure 2 As shown, it consists of a writer, a reader, and a transactional read-write module (MEMORYLOGFILE). There is a data page buffer in the memory of the writer. The data page buffer includes a buffer header and a The data record area of ​​the actual data record, wherein the buffer header includes a 4-byte record number field for storing the record number in the current page buffer, and the transactional read-write module (MEMORYLOGFILE) has a built-in timer device, used to trigger the refresh of the data page buffer.

[0066] In this embodiment, because the fixed length of the data record is limited to 512 bytes, and the maximum delay of the data record from writing to opening and reading cannot exceed 5t, when the read-write system is set, the data page (WP ) is 1000 records, the length of the data page is about 512KB (4+512*1000), and the timing interval for trigg...

Embodiment 2

[0115] The present invention can also be used for high-efficiency reading and writing of large-capacity data with non-fixed-length records limited by the maximum length of the records. For the situation of non-fixed-length records, there are generally two ways to realize it.

[0116] The first method only needs to make appropriate modifications to the data page buffer (such as Image 6 As shown), the effect of Embodiment 1 can be realized. Assuming that the maximum length of a data record is 512 bytes, a fixed length of 512 bytes is still allocated to each data record in the data page buffer, and a field is added to the head of the data record to record each data record The actual length of other data writing and reading logic is the same as that of Embodiment 1. The advantage of using this method is that the logic is simple, and the characteristics of fixed-length records can be used to realize the rapid positioning of records in the data page. In the use of data files and A...

Embodiment 3

[0119] The system adopts multiple hosts and shared storage devices to implement the active-standby architecture. The more common mode is that the business process on the Master host continuously drops business data to the shared storage device. Once the Master host fails or the business process on the Master host is abnormal, The Slave host will use the business process on the machine to take over, and is responsible for loading business data files from the shared storage, and performing subsequent business processing after loading. The realization of this kind of active / standby switchover requires that the business data files are guaranteed to be correct and complete through transactions on the one hand, and on the other hand requires that the takeover of the standby machine can be completed in a relatively short period of time, and the ability to provide business processing to the outside world can be restored as soon as possible. Function.

[0120] Using the method of the p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of electronic information and electronic information data operation, and particularly relates to transaction-based large-volume data read and write methods. According to the write method, data records are organized by virtue of data pages and then are subjected to batch processing, and simultaneously, a great deal of transactional write operation is converted into non-transactional write operation, so that the data volume of transactional read and write is reduced effectively; and by virtue of transaction characteristics of index files, the integrity and the transaction of data are ensured, and the efficient read-write of the mass data is achieved. According to the reading method, a data page index is established by a shared memory, and the quick positioning of the data records is realized by using a bisection method. In an occasion that the mass data need to be subjected to the transactional read-write, the transaction-based large-volume data read and write methods can be used for achieving the aims of reducing the IO (input / output) operation, improving the handing capacity and shortening the time delay.

Description

[technical field] [0001] The invention relates to the technical field of electronic information and electronic information data operation, in particular to a method for reading and writing large-capacity data based on transactions, which is suitable for occasions where rollback, large-volume, and efficient data reading and writing needs to be realized. [Background technique] [0002] In the securities industry, the background of the trading system of the stock exchange is the core of the operation of the entire securities market. During the trading hours, the background of the trading system will receive a large number of market trading orders, and push the results back to the market after processing. In this process, a large number of data records will be generated. For example, when receiving an order, an order confirmation record will be generated; matching A transaction will generate 2 transaction confirmation records (1 buy record and 1 sell record), and these different...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30G06F3/06
Inventor 吴征武剑锋王泊陈晨朱立刘凯刘经纬黄寅飞陆素源郑刚白硕
Owner 上交所技术有限责任公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap