A real-time data transmission method based on server shared disk

Abstract

The application discloses a kind of real-time data transmission methods based on server shared disk, the server transmission architecture that is used includes: subscription server: the server will distribute all generated real-time transaction data to each subscription real-time transaction data client server;Subscription client server: the server will initiate subscription registration to subscription server;First time receives real-time transaction data;Disk monitoring server: the real consumer of real-time transaction data in external network;Through monitoring the data file of local disk, the production and change of real-time data are monitored;Shared network disk: the disk is virtualized network disk, and is mounted connection through windows remote control connection and real physical disk in disk monitoring server.The application uses more simple and efficient server architecture, more stable and high-performance software architecture and more simple and perfect data definition method, and can efficiently transmit a large amount of real-time transaction data to external cluster.

Description

Technical Field

[0001] This invention belongs to the field of real-time data transmission and processing technology, and specifically relates to a real-time data transmission method for Windows clusters based on server shared disks. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, humanity is quickly moving from the IT era to the DT (Data Technology) era. The rise of big data technology has broken down information silos and enabled humanity to uncover more valuable things and information.

[0003] The demand for real-time data processing is growing rapidly, as evidenced across various industries. However, we also see a growing incompatibility between the real-time data processing needs of different industries and enterprises and the current project development methods and supporting tools. Therefore, introducing advanced concepts such as data warehousing, data mining, and OLAP, and using real-time data application platforms to handle massive amounts of information and perform real-time analysis and processing to overcome the difficulties in data processing, is now an essential course for enterprises, internet companies, financial institutions, and government departments.

[0004] For large-scale financial trading systems with a large user base, real-time trading data is undoubtedly the most valuable resource. Real-time trading data reflects current market buying and selling trends, and indirectly represents stock price fluctuations. Furthermore, users can perform further analysis and statistical analysis of the data to make better investment and trading decisions.

[0005] Furthermore, stock trading systems generally have extremely high levels of security management and control. Most of them are located in internal clusters isolated from the external network.

[0006] Clearly, in the context of stock trading systems, acquiring trading data in real-time, encrypting and decrypting it in a relatively efficient and secure manner, analyzing and statistically processing the data, and then transmitting the processed data to the user's end as soon as possible are all crucial for financial traders' trading decisions.

[0007] In a server cluster using Windows as the unified system, data transmission has extremely high security requirements. The internal and external clusters are network isolated, making it impossible to establish either a two-way or one-way network whitelist. Servers within the internal cluster can communicate with each other but cannot access the external cluster or the public internet. Servers within the external cluster can communicate with each other and access the public internet. However, access to the external cluster is only possible via a VPN, establishing a network tunnel to a specific server within the internal cluster. Through this tunnel, only one-way access is possible: external cluster servers access the specific server within the internal cluster via Windows Remote Desktop. Summary of the Invention

[0008] In order to overcome the defects and deficiencies in the prior art, this invention provides a method for transmitting real-time transaction data based on a server shared disk. This method employs a simpler and more efficient server architecture, a more stable and high-performance software architecture, and a more streamlined and complete data definition method, which can efficiently transmit large amounts (TB-level) of real-time transaction data to an external cluster.

[0009] This invention provides the following technical solution: a method for transmitting real-time transaction data based on a server shared disk.

[0010] The server transmission architecture used includes:

[0011] Subscription server: The producer of real-time transaction data. This server distributes all the real-time transaction data it produces to each client server that subscribes to the real-time transaction data.

[0012] Subscription client server: The first consumer of real-time transaction data; this server initiates subscription registration with the subscription server; it receives real-time transaction data immediately; this server is also the only bridge connecting the internal cluster and the external cluster.

[0013] Disk monitoring server: The true consumer of real-time transaction data on the external network; it monitors the production and changes of real-time data by monitoring the data files on the local disk.

[0014] Shared network disk: This disk is a virtualized network disk that is mounted to a real physical disk in the disk monitoring server via Windows remote control connection;

[0015] The software architecture adopted is centered around stock-consumer and stock-analysis. The former is responsible for consuming and transmitting intranet data and writing it to a shared network disk, while the latter is responsible for monitoring, consuming and analyzing extranet data, distributing the processed data to the storage database and WS (WSServer) service, and then pushing it to the actual subscribers by the WS service.

[0016] Preferably, a subscription client is established on the internal network to subscribe to real-time transaction data from the real-time transaction server; the subscription client processes, encrypts, and calculates the received subscription data through a program, and writes it directly to the virtual drive letter mounted on the Windows shared network disk in a format defined by the specified data definition method;

[0017] Data will be written to a virtual disk in the form of files, and the virtual disk is mounted to an external server, namely the disk monitoring server, which monitors all the real physical disks of the local machine through a program, thereby receiving real-time transaction data from the intranet.

[0018] The disk monitoring server reads and analyzes the monitored files, and pushes the data to the user's device and the persistent database in real time.

[0019] Preferably, the virtual drive letter mounted by the Windows shared network disk refers to: when establishing remote control on an external server, the local disk of the current host is set to be mounted; thus, a network disk that can share data will appear on the target intranet server; access is defined in the form of a virtual drive letter.

[0020] Preferably, the subscription client program needs to subscribe to the real-time transaction data of the transaction server. The program needs to monitor and receive the transaction data issued by the transaction server, perform basic cleaning and verification of the transaction data, cache the data in its own memory queue, start multi-threaded consumption of the memory queue, and write the data to the virtual disk file according to the prescribed data definition method.

[0021] Preferably, the disk monitoring server program needs to monitor the data written by the subscription client program, and decrypt, verify, and analyze the data, organize it into standard structured data, write it to the persistent database, and push this real-time structured data to the user.

[0022] Preferably, the format of the specified data definition method refers to the following: when defining data, the batch, time, and completion status are indicated by the file name; the rules for the file content are stock code, price, buy / sell direction, and quantity.

[0023] Preferably, the filename 202212250945-1-87689@@end is separated by "-", where "202212250945" represents 9:45 AM on December 25, 2022, "1" represents the first batch of data in this time period, "87689" is a string of random numbers to avoid duplicate filenames, and "@@end" indicates that the batch of data has been transmitted and informs the parsing program that it can be processed in its entirety.

[0024] Preferably, once a stock code with a single string appears, it represents the detailed data of that stock. Subsequent data represents the current time batch of that stock, including the price, buy / sell direction, and quantity corresponding to the time batch defined in the file name, until the code of the next stock appears, which then marks the beginning of the next stock's data.

[0025] Preferably, real-time synchronization and push to the user end are achieved using the WS protocol for real-time subscription and retrieval. The disk monitoring server program pushes data to the WS server immediately; the server then pushes the data to each subscribed client.

[0026] Preferably, standard structured data is defined, with the following fields: id: primary key identifier; securityCode: stock code; ddate: date; dtime: time; saleCount: number of shares sold; buyCount: number of shares bought.

[0027] This invention employs a simpler and more efficient server architecture, a more stable and high-performance software architecture, and a more streamlined and comprehensive data definition method, which can efficiently transmit large amounts (TB-level) of real-time transaction data to an external cluster. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the server architecture in this invention;

[0029] Figure 2 This is a schematic diagram of the real-time software architecture in this invention;

[0030] Figure 3 This is a schematic diagram illustrating the mounting of a local disk in this invention;

[0031] Figure 4 This is a schematic diagram of the virtual disk mounted in this invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] A method for transmitting real-time transaction data based on a server-shared disk mainly revolves around three core aspects: server architecture, software architecture, and data definition method.

[0035] A simpler and more efficient server architecture is adopted: Instead of a complex server cluster architecture, a streamlined "3+1" server transmission architecture is used. Here, "3" represents the three main servers carrying the data links, such as... Figure 1 As shown. Includes:

[0036] Subscription server: The producer of real-time transaction data. This server distributes all the generated real-time transaction data to each client server that subscribes to the real-time transaction data.

[0037] Subscription client server: The first-hand consumer of real-time transaction data. This server initiates a subscription registration with the subscription server and receives real-time transaction data immediately. This server also serves as the sole bridge connecting the internal and external clusters.

[0038] Disk monitoring server: The actual consumer of real-time transaction data on the external network. It monitors the production and changes of real-time data by monitoring the data files on the local disk.

[0039] The "1" represents the most critical hub in the link—the shared network disk: this disk is a virtualized network disk that is mounted and connected to the real physical disk in the disk monitoring server via Windows remote control connection.

[0040] Adopt a more stable and high-performance software architecture: such as Figure 2 As shown, the core components are stock-consumer and stock-analysis. The former is responsible for consuming and transmitting internal network data, writing it to a shared network disk. The latter is responsible for monitoring, consuming, and analyzing external network data. It then distributes the processed data to the storage database and the web service, which in turn pushes it to the actual subscribers.

[0041] A more streamlined and comprehensive data definition method is adopted: To achieve more efficient transmission and avoid wasting space due to duplicate fields, the data definition uses filenames to indicate the batch, time, and completion status. For example, the filename is: 202212250945-1-87689@@end. Separated by "-", "202212250945" represents 9:45 AM on December 25, 2022; "1" indicates the first batch of data within that time period; "87689" is a string of random numbers to avoid duplicate filenames; and "@@end" indicates that the batch of data has been transmitted successfully, informing the parsing program that it can be processed in its entirety. The file content follows the rules of stock code, price, buy / sell direction (0 or 1 represents buy or sell), and quantity, as shown in the table below. Once a single-character stock code appears, it represents the detailed data for that stock; subsequent data represents the price, buy / sell direction, and quantity for the current time batch (corresponding to the time batch defined in the filename). This continues until the next stock code appears, marking the start of the next stock's data.

[0042] Example data is shown in Table 1.

[0043]

[0044] Table 1 Data Definition Method

[0045] To better illustrate the design of the method, let's further explain the overall data flow: Data is initially generated and distributed from the subscription server. The stock-consumer service, located on the subscription client server, has already proactively registered its data subscription request with the subscription server. Therefore, the moment the subscription server generates data, it sends the data to the subscription client server. The stock-consumer receives this data, performs simple verification and cleaning, and then integrates the data, writes it to a file, and stores the file on a virtual network disk.

[0046] The virtual network disk generates files, and simultaneously, files are generated on the local disk of the disk monitoring server located on the external network. At this time, stock-analysis on the disk monitoring server detects the newly added file data, parses the filename and content according to defined data rules, and then further structures the read data. Finally, the structured data is sent to the database storage and also to the WS service, allowing it to be pushed to the terminals of subscribed users immediately.

[0047] Example 2

[0048] This invention designs a novel transmission architecture. The server architecture is as follows: Figure 1 As shown.

[0049] A subscription client is established on the internal network to subscribe to real-time trading data from the subscription server (real-time trading server). The subscription client processes, encrypts, and calculates the received subscription data through a program, and writes it directly to the virtual drive letter mounted on the Windows shared network disk in a format defined by the specified data definition method.

[0050] Data will be written to a virtual disk as files, and the virtual disk will be mounted to an external server. Figure 1 The disk monitoring server monitors all the actual physical disks on the local machine. The disk monitoring server receives real-time transaction data from the intranet by monitoring the local physical disks through a program.

[0051] The disk monitoring server reads and analyzes the monitored files, and pushes the data to the user's device and the persistent database in real time.

[0052] A Windows shared network disk, when mounted as a virtual drive letter, is configured to mount the local disk of the current host when a remote control is established on an external server. This creates a network disk on the target internal network server that can share data. Access is typically defined as a virtual drive letter. For example... Figure 3 and 4 As shown.

[0053] The subscription client program plays a crucial role in the software architecture. It needs to subscribe to real-time trading data from the trading server. This program monitors and receives trading data from the trading server, performs basic cleaning and validation on the data (checking for null values ​​and valid numbers), and then caches the data in its own memory queue. Multiple threads are then used to consume this memory queue, writing the data to files on the virtual disk according to the defined data definition methods.

[0054] The disk monitoring server program needs to monitor the data written by the subscribed client programs. It then decrypts, verifies, and analyzes the data, organizing it into standard structured data. Simultaneously, this real-time structured data is written to the persistent database and pushed to the user.

[0055] To enable users to access real-time data more effectively and to improve network communication synchronization performance, this method does not use the traditional HTTP mode for data retrieval. Instead, it employs the WS protocol for real-time subscription and retrieval. The disk monitoring server program pushes the data to the WS server as soon as it is available. The server then pushes the data to each subscribed client.

[0056] To facilitate storage and better parsing and display on the user end, standard structured data has been defined. Its defined fields are shown in Table 2.

[0057] Fields name id Field primary key identifier securityCode Stock Code ddate date dtime time saleCount Sales volume buyCount Purchase quantity

[0058] Table 2 Structured Field Definitions

[0059] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A server-based shared disk real-time transaction data transmission method, characterized in that: The server transmission architecture adopted includes: A subscription server: the producer of real-time transaction data, which distributes all generated real-time transaction data to each client server subscribing to real-time transaction data; A subscription client server: the first-hand consumer of real-time transaction data; the subscription client server initiates a subscription registration to the subscription server; real-time transaction data is received at the first time; the subscription client server is also the only bridge connecting the internal cluster and the external cluster; A disk monitoring server: the real consumer of real-time transaction data in the external network; by monitoring the data files of the local disk, the production and changes of real-time data are monitored; A shared network disk: the shared network disk is a virtualized network disk, which is mounted and connected through windows remote control connection and the real physical disk in the disk monitoring server; The software architecture adopted takes stock-consumer and stock-analysis as the core, the former is responsible for the consumption and transmission of internal network data, and writes it into the shared network disk, and the latter is responsible for the monitoring, consumption and analysis of external network data, and distributes the processed data to the storage database and ws service, and then pushes it to the actual subscription user's hand by the ws service; A subscription client is established in the internal network to subscribe to the real-time transaction data of the real-time transaction server; the subscription client processes, encrypts and calculates the received subscription data through a program, and directly writes it into the virtual drive mounted by the Windows shared network disk in the form of a specified data definition method; The data is written in the form of a file to the virtual disk, and the virtual disk is mounted to the external server, i.e. the disk monitoring server, all of which are real physical disks, and the disk monitoring server monitors the local physical disk through a program, thereby receiving real-time transaction data from the internal network; The disk monitoring server reads and analyzes the monitored files and synchronously pushes them to the user end and the persistent database.

2. The method for transmitting real-time transaction data based on server shared disk according to claim 1, characterized in that: The virtual drive mounted by the Windows shared network disk refers to: when establishing remote control on the external server, the local disk of the current host is set to be mounted; and then on the target internal network server, a network disk that can share data appears; It is defined as access in the form of a virtual drive.

3. The method of claim 1, wherein the method further comprises: The program of the subscription client needs to subscribe to the real-time transaction data of the transaction server, and needs to monitor and receive the transaction data issued by the transaction server, and then perform basic cleaning and verification on the transaction data, and then cache the data to the internal memory queue, start multi-threaded consumption of the memory queue, and write the data into the file of the virtual disk according to the specified data definition method.

4. The method of claim 1, wherein the method further comprises: The program of the disk monitoring server needs to monitor the data written by the subscription client program, decrypt, verify and analyze the data, arrange it into standard structured data, write it into the persistent database end, and push the real-time structured data to the user.

5. The method for transmitting real-time transaction data based on server shared disk according to any one of claim 1 or claim 3, characterized in that: The format of the specified data definition method is that, when defining data, the batch, time, and completion are indicated by the file name, and the rules of the file content are the stock code, price, buying and selling direction, and quantity.

6. The method of claim 5, wherein the method further comprises: The file name: 202212250945-1-87689@@end is split by "-" in the file name, where "202212250945" represents 9:45 am on December 25, 2022, "1" represents the first batch of data at that time point, "87689" is a string of random numbers to avoid duplicate file names, and "@@end" indicates that the first batch of data has been transmitted and completed, informing the parsing program that it can be processed in full.

7. The method of claim 6, wherein the method further comprises: Once a one-character stock code appears, it is the detailed data of that stock, and the subsequent data is the current time batch of that stock, i.e., the price, buying and selling direction, and quantity corresponding to the time batch defined in the file name, until the next stock code appears, which is the start of the data of the next stock.

8. The method for transmitting real-time transaction data based on server shared disk according to any one of claim 1 or claim 4, characterized in that: Real-time synchronization is pushed to the user end using the WS protocol for real-time subscription and pulling. The disk monitoring server program will push the data to the WS service end in the first time, and the service end will push the data to each subscribed client.

9. The method of claim 4, wherein the method further comprises: The standard structured data is defined, and the defined fields are as follows: id: field primary key identifier; securityCode: stock code; ddate: date; dtime: time; saleCount: sell quantity; buyCount: buy quantity.

Images