A method of quickly loading large data groups based on dss data system

Abstract

The invention discloses a DSS-based efficient data system design method. A large virtual memory is opened up and a large amount of physical memories are submitted to a server for use; hardware resources of the server are fully utilized; by use of the virtual memory, the loading of a large data group becomes possible and the memory opening-up efficiency is remarkably improved; the data access is convenient and quick; by locking the physical memories, memory page exchange in a process space of the server during operation is prevented to the maximum extent, so that the performance loss of the server is greatly reduced; and by use of the virtual memory, one-time loading of the large data group is realized, so that the situation of reloading in an existing scheme is avoided, and especially the condition that the data loading situation in need of an I / O process causes crash on the server is avoided.

Description

technical field [0001] The invention relates to a method for quickly loading large data groups based on a DSS data system. Background technique [0002] Darwin Streaming Server is referred to as DSS. DSS is an open source real-time streaming media server program provided by Apple. The whole program is written in pure C++. It is designed to follow the programming principles of high performance, simplicity, and modularity, so as to achieve high efficiency and good scalability. Therefore, DSS The server system is quite ideal in terms of cross-platform support. It can run on Windows NT, Windows 2000 and above Windows kernel versions, and can also run well on various versions of *NIX, including Mac OS X, Linux , FreeBSD, Solaris, and DSS is developed based on the standard streaming media protocol RTSP, RTP / RTCP, so it is extremely extensive and versatile in use. [0003] The current data system based on the DSS server has the following defects: [0004] 1. The "chimney" archit...

AI Technical Summary

Problems solved by technology

[0004] 1. The "chimney" architecture of traditional IT business deployment leads to high configuration and low usage of equipment hardware. Considering the future development of application system business and sudden business needs, in order to meet the performance and capacity carrying requirements of applications, users often choose There will be a certain percentage of margin in the configuration of hardware devices such as computing, storage, and network, but after the hardware resources are online, the load of the application system will not be too high for a certain period of time, which makes the hardware devices with higher configuration Utilization rate is not high

[0005] 2. The existing DSS server data system stores data and uses the heap as the main means of exchange. In this case, if the amount of data is small, it will have little impact on the case of infrequent data use. However, for enterprise commercial use For high-level servers, it is often necessary to load a large amount of data and frequently operate large data groups. If you use the original heap operation method, you need to open up and release space every time, and the open space is very large, and during operation A large number of memory fragments will be generated in the system, making the space and time efficiency of memory usage very low, and the program will deteriorate after running for a period of time

[0006] 3. The existing DSS server does not lock the memory pages. The server often has a very large memory capacity and uses it exclusively. However, the real physical memory usage in the actual operation process is very low, and page faults frequently occur when operating large data groups. , and the generation of page fault interruption will seriously affect the response speed of the server and then accumulate data access, and eventually even cause the server to go down, and this kind of problem is often difficult to troubleshoot

[0007] 4. For the I / O of large data groups, the existing application based on the DSS framework often uses small batches to read and write large amounts of data from the database, and the method of loading when needed is very limited in use, and for Frequent I / O operations are unacceptable consumption in server usage

Images