Two-stage-system-based distributed data compression processing method
A distributed data and compression processing technology, applied in transmission systems, electrical components, etc., can solve the problems of increased server memory consumption, server CPU burden, network resource consumption, etc., to reduce memory load, respond quickly, and save network resources. Effect
Active Publication Date: 2012-06-13
BEIJING HUADIAN TIANREN ELECTRIC POWER CONTROL TECH
View PDF2 Cites 10 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Imagine a database with an upper limit requirement of 100,000 label points. The data is updated once per second, and the data volume is 781kb. If all are uploaded to the server for compression, it will take up more than 6M of bandwidth, which greatly consumes network resources.
In addition, the data of each tag point can only be compressed using the compression algorithm after the server has accumulated a certain amount of data, which will greatly increase the memory consumption of the server
Even if the server memory is sufficient, such a large-scale data compression process will bring a great burden to the server CPU.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0022] The technical solution of the present invention will be described in further detail below in conjunction with the accompanying drawings.
[0023] Such as figure 1 Shown is a schematic diagram of the distributed compression processing method disclosed in the present invention.
[0024] The interface machine is responsible for collecting data from the underlying control system (such as DCS, PLC, etc.), and writing data into the database in the server by calling the database interface API. Due to the massive amount of data and the timeliness of data collection, the interface machine needs to quickly write the collected data to the server. The TCP / IP protocol is used between the interface machine and the server, and the transmission rate of the network has also become an important factor restricting the writing speed of the database. Therefore, compress the data on the interface machine first, and then upload it to the server to reduce the impact of the network environmen...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a two-stage-system-based distributed data compression processing method. Based on the conventional mode of compressing data in a server, the data is compressed in interface machines, the compressed data is uploaded to the server after a certain time or when a certain amount of compressed data is obtained, and after the uploaded data is accumulated to a certain amount, the server calls a file processing program to store the data. The compression load of the server is transferred to a plurality of interface machines, so that the memory load of the server is decreased, unnecessary overhead is decreased, the server can give a quicker response to real-time data, and resources of each node in a network are rationally utilized to realize the rational configuration of the resources. By the compression mode of compressing the data in the interface machines and then uploading the compressed data to the server, data transmission in the network is greatly reduced, and network resources are saved. The method for compressing the data in the interface machines and then uploading the compressed data to the server is more suitable for the realization of a real-time database with a super high data volume.
Description
technical field [0001] The application belongs to the technical field of data compression in real-time historical databases, and specifically relates to a distributed data compression processing method based on a two-level system. Background technique [0002] The real-time historical database product is the basic platform of enterprise informatization and the bridge of management and control integration. A large amount of real-time data generated in the production process is one of the valuable resources of process enterprises. These data need long-term storage, fast retrieval, and serve as the basis for production data analysis, data mining, optimization control and optimization management. A typical process enterprise needs to integrate data collection points that usually range from thousands to hundreds of thousands. Since the data at the collection points is dynamically changing and the refresh rate is fast (second level), it is difficult to store such large-scale mass...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): H04L29/08
Inventor 蒋禾青黄孝彬程睿君康芳
Owner BEIJING HUADIAN TIANREN ELECTRIC POWER CONTROL TECH