Ordered file collection and breakpoint resume method

Abstract

The invention discloses a method for orderly file collection and breakpoint resume, which comprises the following steps of: file classification: classifying different types of files by using prefix time and suffix, and classifying the files with the same prefix into the same type of files; file storage; the server side classifies and stores files according to the ip address of the application module and the name of the application module, namely a gateway, a transaction engine and the like; the agent end and the server end are interacted through a tcp protocol; the problems that in the prior art, orderliness of data files is not concerned, continuous transmission of a single file is only considered in the breakpoint continuous transmission process, and data time sequence chaos is caused are solved.

Description

Technical Field

[0001] The invention is used in the field of data collection and monitoring, and specifically is a method for orderly file collection and breakpoint resume transmission. Background Art

[0002] In the existing financial industry, the core transaction module will generate a lot of logs and data. These data are of key significance to the status of the entire financial module. Therefore, the integrity and order of data must be guaranteed during the collection and transmission process. Interruptions will generate breakpoint information. The integrity and order of data must be restored to avoid the monitored state being chaotic. Breakpoint information is the metadata of related files and the location of the transmission disconnection after the full data transmission is interrupted. Existing data collection does not pay attention to the orderliness of data files. The breakpoint resume process only considers the resume of a single file, which will cause data timing confusion. Summary of the invention

[0003] The purpose of the present invention is to solve the problem that the prior art does not pay attention to the orderliness of data files, and the breakpoint resume process only considers the resume of a single file, which will cause data time sequence confusion.

[0004] In order to solve the above problems, the present invention provides a method for orderly file collection and breakpoint resume, and the specific steps are as follows:

[0005] S1: Document classification;

[0006] S1-1: Different types of files are classified using prefix_time.suffix. Files with the same prefix are of the same type.

[0007] The prefixes include: application module name A, application module name B, application module name C;

[0008] The time includes: for example, 202409201306, which means the file was generated at 13:06 on September 20, 2024;

[0009] Suffixes include: .sh, .pl, py;

[0010] Classification is to reflect the relationship between multiple files output by the same application module. Only files of the same type will be sorted by time and transmitted in sequence. Files of different types can be mixed and transmitted without affecting each other.

[0011] S2: file storage;

[0012] S2-1: The server classifies and stores files according to the IP address and name of the application module, i.e., gateway and transaction engine. The storage hierarchy is: storage root directory / IP address / application module name / file.

[0013] An application module is a program used to complete a certain function. The specific function is not limited. When completing the function, it usually stores newly generated data in the form of files. The data content is not limited. The functions of this data include but are not limited to: 1. Serving as input for other modules or 2. Recording the current application module status or 3. Recording events that occur when the process is running.

[0014] When storing files, a folder hierarchy is required. This hierarchy is also called a "path". The classified storage here is to determine the storage path of the file. The classification of file names above is to distinguish files in the same folder. The classification by path here is to be able to clearly determine which machine the file comes from and which module it is generated by.

[0015] S3: the agent interacts with the server;

[0016] The agent is a program used to collect part or all of the files generated by the application module, and the collected data is sent to the server.

[0017] The server is a program that stores data transmitted by the agent to disk.

[0018] Both agent and server are ordinary TCP interactive programs and can be created by yourself.

[0019] S3-1: The server actively obtains the IP information of the agent through a TCP connection;

[0020] S3-2: the server provides the agent with the breakpoint information of the received file of the IP where the agent is located, thereby completing the interaction;

[0021] The breakpoint information includes the file name and the size of the file that has been transferred, where the "size of the file that has been transferred" is called the breakpoint.

[0022] S3-3: When transmitting files after interaction, the agent carries the application module name and the file name in the data and sends it to the server;

[0023] S3-3-1: The agent sends files as follows:

[0024] S3-3-2: the agent sends the files of the same type to be transmitted in order according to the time in the file name;

[0025] If there is breakpoint information in the same type of file to be transmitted, it means that the file has been transmitted before, then the file is transmitted starting from the breakpoint, that is, data is continuously written from the end of the file;

[0026] If there is no breakpoint information in the same type of file to be transmitted, that is, the server does not have the information of the file, it means that this is a file that has not been transmitted, and the transmission starts from the beginning.

[0027] In a preferred manner, different types of files in step S3-3-2 can be sent simultaneously through different channels.

[0028] In a preferred manner, the file is generated by the application module, and the specific content can be any data, that is, binary data or text data.

[0029] The present invention can achieve the following beneficial effects: orderly, real-time and complete monitoring and transmission of important events and performance indicators of a trading system are realized, and the correct monitoring of the trading system is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the interaction process of the present invention;

[0031] Figure 2 It is a detailed interactive flow diagram of the present invention. DETAILED DESCRIPTION

[0032] Embodiment 1:

[0033] The detailed interaction process of this system is as follows Figure 2 . First, start the agent and server. There is no requirement for the order of starting. When the agent starts, it first reads the configuration file, which records the directories to be monitored and the file prefixes and suffixes to be monitored. These configurations will be saved in the agent's memory for query and comparison. When the server starts, it first traverses the storage root directory and saves the obtained file record results in the form of breakpoint information in the memory. The agent connects to the server. If the connection is unsuccessful, it waits and reconnects in a loop. If the connection is successful, the server queries the breakpoint information of the agent's corresponding IP and sends it to the agent. The agent receives the breakpoint information, traverses the local monitoring directory according to the configuration information, and forms a file list by type. Use the breakpoint information to compare with the file list, and ignore the breakpoint information of the file that does not match the file. According to the configured monitoring directory, the agent starts a coroutine (lightweight thread) for each directory to monitor the file. If there is breakpoint information, it starts the transmission from the breakpoint of the file, and if there is no breakpoint information, it starts the transmission from the first file of the type. If the maximum position of the current file has been transmitted, it monitors whether there is new file content written to the file or a new file is created, and continues the transmission. At this time, the server reads and stores the monitoring files completely in chronological order.

[0034] Embodiment 2:

[0035] A method for orderly file collection and breakpoint resuming, the specific steps are as follows:

[0036] S1: Document classification;

[0037] S1-1: Different types of files are classified using prefix_time.suffix. Files with the same prefix are of the same type.

[0038] S2: file storage;

[0039] S2-1: The server classifies and stores files according to the IP address and name of the application module, i.e., gateway and transaction engine. The storage hierarchy is: storage root directory / IP address / application module name / file.

[0040] S3: the agent interacts with the server;

[0041] S3-1: The server actively obtains the IP information of the agent through a TCP connection;

[0042] S3-2: the server provides the agent with the breakpoint information of the received file of the IP where the agent is located, thereby completing the interaction;

[0043] S3-3: When transmitting files after interaction, the agent carries the application module name and the file name in the data and sends it to the server;

[0044] S3-3-1: The agent sends files as follows:

[0045] S3-3-2: the agent sends the files of the same type to be transmitted in order according to the time in the file name;

[0046] If there is breakpoint information in the same type of file to be transmitted, it means that the file has been transmitted before, then the file is transmitted starting from the breakpoint, that is, data is continuously written from the end of the file;

[0047] If there is no breakpoint information in the same type of file to be transmitted, that is, the server does not have the information of the file, it means that this is a file that has not been transmitted, and the transmission starts from the beginning.

[0048] In step S3-3-2, files of different types can be sent simultaneously through different channels.

[0049] The file is generated by the application module, and the specific content can be any data, that is, binary data or text data.

[0050] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for orderly file collection and breakpoint resuming, characterized in that: The specific steps are as follows: S1: Document classification; S1-1: Different types of files are classified using prefix_time.suffix. Files with the same prefix are of the same type. S2: file storage; S2-1: The server classifies and stores files according to the IP address and name of the application module, i.e., gateway and transaction engine. The storage hierarchy is: storage root directory / IP address / application module name / file. S3: the agent interacts with the server; S3-1: The server actively obtains the IP information of the agent through a TCP connection; S3-2: the server provides the agent with the breakpoint information of the received file of the IP where the agent is located, thereby completing the interaction; S3-3: When transmitting files after interaction, the agent carries the application module name and the file name in the data and sends it to the server; S3-3-1: The agent sends files as follows: S3-3-2: the agent sends the files of the same type to be transmitted in order according to the time in the file name; If there is breakpoint information in the same type of file to be transmitted, it means that the file has been transmitted before, then the file is transmitted starting from the breakpoint, that is, data is continuously written from the end of the file; If there is no breakpoint information in the same type of file to be transmitted, that is, the server does not have the information of the file, it means that this is a file that has not been transmitted, and the transmission starts from the beginning.

2. The method for orderly file collection and breakpoint resume according to claim 1, characterized in that: In step S3-3-2, files of different types can be sent simultaneously through different channels.

3. The method for orderly file collection and breakpoint resume according to claim 1, characterized in that: The file is generated by the application module, and the specific content can be any data, that is, binary data or text data.

Images