Data processing method and device, computer equipment and computer readable storage medium

Abstract

The present invention discloses a data processing method and device, a computer device and a computer-readable storage medium, and relates to the technical field of data preprocessing. The method comprises: according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database, parsing the source system data using the parsing program corresponding to the source system data, and obtaining target format data. The present invention improves the flexibility of processing different source data by pre-configuring the parsing program in the parsing program database, and at the same time, uses the parsing program corresponding to the source system data to parse the source system data, thereby obtaining target format data with a unified format, and improving the efficiency of processing different source data.

Description

Technical Field

[0001] The present invention relates to the technical field of data preprocessing, and in particular to a data processing method and apparatus, a computer device, and a computer-readable storage medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] Data analysis refers to the use of appropriate statistical analysis methods to analyze a large amount of collected data, summarize, understand and digest them, in order to maximize the development of data functions and give full play to the role of data. Data analysis is the process of detailed study and summary of data in order to extract useful information and form conclusions.

[0004] In order to meet regulatory compliance and basic internal risk reporting requirements, financial market risks need to centralize and integrate fund transaction business data and market data, and analyze the data to obtain risk measurement results and achieve end-of-day limit management. Market data accessed from the outside includes multiple data sources, such as WIND, POMS, BLOOMBERG, REUTER, and KONDOR data. Among them, the data of POMS, BLOOMBERG, REUTER, and KONDOR are all stored in the form of files, and the organization of the files is different. In addition, the data formats of multiple data sources are not unified, which affects the efficiency of subsequent data analysis.

[0005] Therefore, the existing technology has problems of poor flexibility and low efficiency in data processing due to the non-uniform data formats of multiple data sources. Summary of the invention

[0006] An embodiment of the present invention provides a data processing method for improving the flexibility and efficiency of processing different source data. The data processing method includes:

[0007] According to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database;

[0008] The source system data is parsed using the parsing program corresponding to the source system data to obtain the target format data.

[0009] An embodiment of the present invention further provides a data processing device for improving the flexibility and efficiency of processing different source data. The data processing device includes:

[0010] A parsing program acquisition module, used to acquire a parsing program corresponding to the source system data from a pre-configured parsing program database according to the received source system data;

[0011] The data parsing module is used to parse the source system data using the parsing program corresponding to the source system data to obtain the target format data.

[0012] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the computer program.

[0013] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above data processing method.

[0014] In the embodiment of the present invention, according to the received source system data, the parsing program corresponding to the source system data is obtained from the pre-configured parsing program database, and the source system data is parsed using the parsing program corresponding to the source system data to obtain target format data. In the embodiment of the present invention, the flexibility of processing different source data is improved by pre-configuring the parsing program in the parsing program database, and the source system data is parsed using the parsing program corresponding to the source system data, thereby obtaining target format data with a unified format, and improving the efficiency of processing different source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0016] Figure 1 A flowchart of a data processing method according to a first embodiment of the present invention;

[0017] Figure 2 A flowchart of a data processing method according to a second embodiment of the present invention;

[0018] Figure 3 A flowchart of a data processing method according to a third embodiment of the present invention;

[0019] Figure 4 A flowchart of a data processing method according to a fourth embodiment of the present invention;

[0020] Figure 5 A flowchart for implementing step 401 in the data processing method provided in the fifth embodiment of the present invention;

[0021] Figure 6A flowchart for implementing step 501 in the data processing method provided in the sixth embodiment of the present invention;

[0022] Figure 7 A flowchart of a data processing method according to a seventh embodiment of the present invention;

[0023] Figure 8 A flowchart of a data processing method according to an eighth embodiment of the present invention;

[0024] Fig. 9 A flowchart of a data processing method according to a ninth embodiment of the present invention;

[0025] Fig.10 A flowchart of a data processing method according to a tenth embodiment of the present invention;

[0026] Fig.11 A functional module diagram of a data processing device provided for the eleventh embodiment of the present invention;

[0027] Fig.12 A functional module diagram of a data processing device provided for the twelfth embodiment of the present invention;

[0028] Fig.13 A structural block diagram of a data processing device provided for the thirteenth embodiment of the present invention;

[0029] Fig.14 A structural block diagram of a data processing device provided for a fourteenth embodiment of the present invention;

[0030] Fig.15 A structural block diagram of a Req file generating unit 1401 in a data processing device provided in a fifteenth embodiment of the present invention;

[0031] Fig.16 A structural block diagram of a first Req file generating unit 1501 in a data processing device provided in a sixteenth embodiment of the present invention;

[0032] Fig.17 A structural block diagram of a data processing device provided for a seventeenth embodiment of the present invention;

[0033] Fig.18 A structural block diagram of a data processing device provided for an eighteenth embodiment of the present invention;

[0034] Fig.19 A structural block diagram of a data processing device provided for the nineteenth embodiment of the present invention;

[0035] Fig. 20 This is a structural block diagram of a data processing device provided in the twentieth embodiment of the present invention. DETAILED DESCRIPTION

[0036] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0037] Figure 1 The implementation flow of the data processing method provided by the first embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0038] like Figure 1 As shown, a data processing method comprises:

[0039] Step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database;

[0040] Step 102: parse the source system data using a parsing program corresponding to the source system data to obtain target format data.

[0041] When processing data from different source systems into target format data with a unified format, by pre-configuring parsing programs corresponding to different source system data to form a parsing program database, when receiving source system data, based on the received source system data, the parsing program corresponding to the source system data is obtained from the pre-configured parsing program database to improve the efficiency of obtaining the parsing program, thereby improving the efficiency of processing source system data in different formats. Finally, the source system data is parsed using the parsing program corresponding to the source system data to obtain the desired target format data. Among them, the source system data and the target format data are adjusted and modified by technicians in this field according to actual needs to improve the flexibility of processing source system data in different formats.

[0042] For example, when the first source system data is received, the first parsing program corresponding to the first source system data is obtained from the pre-configured parsing program database based on the received first source system data, and the first source system data is parsed using the first parsing program corresponding to the first source system data to obtain the desired target format data. When the second source system data is received, the second parsing program corresponding to the second source system data is obtained from the pre-configured parsing program database based on the received second source system data, and the second source system data is parsed using the second parsing program corresponding to the second source system data to obtain the desired target format data. In this way, both the first source system data and the second source system data can be converted into target format data. In a certain embodiment of the present invention, after obtaining the target format data of different source systems, the target format data of different source systems are uploaded to the data warehouse library so that the data warehouse can perform data analysis on the target format data of different source systems, etc.

[0043] In the embodiment of the present invention, according to the received source system data, the parsing program corresponding to the source system data is obtained from the pre-configured parsing program database, and the source system data is parsed using the parsing program corresponding to the source system data to obtain the target format data. The embodiment of the present invention improves the flexibility of processing different source data by pre-configuring the parsing program in the parsing program database, and at the same time, the source system data is parsed using the parsing program corresponding to the source system data, thereby obtaining the target format data with a unified format, and improving the efficiency of processing different source data.

[0044] Figure 2 The implementation flow of the data processing method provided by the second embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0045] In one embodiment of the present invention, in order to improve the flexibility of the parsing program configuration, as Figure 2 As shown in the above Figure 1 Based on the method steps shown, the data processing method also includes:

[0046] Step 201: configure the parsing program in the parsing program database according to the received configuration instruction.

[0047] When configuring the parsing program in the parsing program database, the parsing program in the parsing program database may be configured based on the received configuration instruction. In addition, the parsing program in the configured parsing program database may be modified based on the received modification instruction to further improve the flexibility of the parsing program configuration. For example, the number of parsing programs in the parsing program database or the parameters of the parsing programs in the parsing program database may be modified, and the modification may include adding or deleting the parsing program, and may also include adjusting and replacing the parameters of the parsing program, etc.

[0048] In the embodiment of the present invention, the parsing program in the parsing program database is configured according to the received configuration instruction, so that the flexibility of the parsing program configuration can be improved.

[0049] Figure 3 The implementation flow of the data processing method provided by the third embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0050] In one embodiment of the present invention, the source system data includes Poms system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Poms system data, Figure 3 As shown, in the data processing method, step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database includes:

[0051] Step 301, obtaining a file to be parsed containing Poms system data from a directory of files to be parsed of the Poms system;

[0052] Step 302, according to the acquired file to be parsed containing the Poms system data, a Poms data parsing program corresponding to the Poms system data is acquired from a pre-configured parsing program database;

[0053] Step 102, using a parsing program corresponding to the source system data to parse the source system data to obtain target format data, including:

[0054] Step 303: Invoke the Poms data parsing program corresponding to the Poms system data through a command line to parse the Poms system data and obtain UNIX format data.

[0055] Specifically, when the source system data includes Poms system data and the target format data includes UNIX format data, the to-be-parsed file containing the Poms system data is obtained from the to-be-parsed file directory Blp of the Poms system, for example, the to-be-parsed file containing the Poms system data is copied to the Poms system intermediate directory Mid.

[0056] Then, according to the obtained file to be parsed containing Poms system data, the Poms data parsing program corresponding to the Poms system data is obtained from the parsing program database, and the Poms data parsing program is called by the command line to parse the Poms system data, and the parsed file is stored in the Poms system result directory DEAL. The parsed file format is DOS format, and the parsed DOS format file is converted into target format data, that is, UNIX format data. Before calling the Poms data parsing program to parse the Poms system data, it can also include filtering invalid data in the Poms system data, calling the Poms data parsing program to parse the Poms system data, including calling the Poms data parsing program to parse the Poms system data after filtering out the invalid data.

[0057] Table 1 below shows the configuration files and their functions required for parsing Poms system data to obtain UNIX format data using the Poms data parsing program:

[0058]

[0059] Table 1 Configuration files and their functions required to parse Poms system data to obtain UNIX format data

[0060] In an embodiment of the present invention, a file to be parsed containing Poms system data is obtained from a directory of files to be parsed of the Poms system, and a Poms data parsing program corresponding to the Poms system data is obtained from a pre-configured parsing program database according to the obtained file to be parsed containing the Poms system data. The Poms system data is parsed by calling the Poms data parsing program corresponding to the Poms system data through a command line to obtain UNIX format data, thereby improving the processing efficiency of the Poms system data.

[0061] Figure 4 The implementation flow of the data processing method provided by the fourth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0062] In one embodiment of the present invention, the source system data includes Bloomberg system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Bloomberg system data, Figure 4 As shown, in the data processing method, step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database includes:

[0063] Step 401, generating a Req data file containing Bloomberg system data;

[0064] Step 402, according to the obtained Req data file containing Bloomberg system data, obtaining a Bloomberg data parsing program corresponding to the Bloomberg system data from a pre-configured parsing program database;

[0065] Step 102, using a parsing program corresponding to the source system data to parse the source system data to obtain target format data, including:

[0066] Step 403: Invoke the Bloomberg data parsing program corresponding to the Bloomberg system data through the command line to parse the Bloomberg system data and obtain UNIX format data.

[0067] Specifically, when the source system data includes Bloomberg system data and the target format data includes UNIX format data, a Req data file including the Bloomberg system data is first generated, and then the Req data file including the Bloomberg system data is downloaded to a to-be-parsed file directory Res of the Bloomberg system, and the to-be-parsed file including the Bloomberg system data is obtained from the to-be-parsed file directory Res of the Bloomberg system, for example, the to-be-parsed file including the Bloomberg system data is copied to an intermediate directory Mid of the Bloomberg system.

[0068] Then, according to the obtained file to be parsed containing Bloomberg system data, the Bloomberg data parsing program corresponding to the Bloomberg system data is obtained from the parsing program database, and the Bloomberg data parsing program is called through the command line to parse the Bloomberg system data, and the parsed file is stored in the Bloomberg system result directory DEAL. Among them, the format of the parsed file is DOS format, and then the parsed DOS format file is converted into the target format data, that is, UNIX format data.

[0069] Before calling the Bloomberg data parsing program to parse the Bloomberg system data, it may also include filtering invalid data in the Bloomberg system data, calling the Bloomberg data parsing program to parse the Bloomberg system data, including calling the Bloomberg data parsing program to parse the Bloomberg system data after filtering out the invalid data.

[0070] Table 2 below shows the configuration files and their functions required for parsing Bloomberg system data to obtain UNIX format data using the Bloomberg data parsing program:

[0071]

[0072] Table 2. Configuration files and their functions required to parse Bloomberg system data to obtain UNIX format data

[0073] In the embodiment of the present invention, a file to be parsed containing Bloomberg system data is obtained from a directory of files to be parsed of the Bloomberg system, and a Bloomberg data parsing program corresponding to the Bloomberg system data is obtained from a pre-configured parsing program database according to the obtained file to be parsed containing the Bloomberg system data. The Bloomberg data parsing program corresponding to the Bloomberg system data is called through a command line to parse the Bloomberg system data to obtain UNIX format data, thereby improving the processing efficiency of the Bloomberg system data.

[0074] Figure 5 The implementation flow of step 401 in the data processing method provided by the fifth embodiment of the present invention is shown. For ease of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0075] In one embodiment of the present invention, the Bloomberg system data includes first type Bloomberg system data and second type Bloomberg system data. In order to improve the accuracy and comprehensiveness of generating the Req data file, as Figure 5 As shown, step 401 generates a Req data file containing Bloomberg system data, including:

[0076] Step 501, generating a Req data file containing first type Bloomberg system data based on UNIX format data obtained by parsing Poms system data using a Poms data parsing program; or

[0077] Step 502: Generate a Req data file containing the second type of Bloomberg system data.

[0078] Bloomberg system data includes first type Bloomberg system data (2 batches of market data, T+1, 04:30) and second type Bloomberg system data (2 batches of T, 21:30 and 2 batches of T+1, 04:50). When generating a Req data file containing the first type Bloomberg system data, it is necessary to use the UNIX format data obtained by parsing the Poms system data by the Poms data parsing program as a basis. When generating a Req data file containing the second type Bloomberg system data, it is not necessary to use the UNIX format data obtained by parsing the Poms system data by the Poms data parsing program as a basis.

[0079] In an embodiment of the present invention, based on UNIX format data obtained by parsing Poms system data by a Poms data parsing program, a Req data file containing first type Bloomberg system data is generated, and a Req data file containing second type Bloomberg system data is generated, which can improve the accuracy and comprehensiveness of the generated Req data file.

[0080] Figure 6 The implementation flow of step 501 in the data processing method provided by the sixth embodiment of the present invention is shown. For ease of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0081] In one embodiment of the present invention, in order to improve the accuracy and efficiency of generating a Req data file containing the first type of Bloomberg system data, as shown in FIG. Figure 6 As shown, step 501, based on the UNIX format data obtained by parsing the Poms system data by the Poms data parsing program, generates a Req data file containing the first type of Bloomberg system data, including:

[0082] Step 601, calling the file acquisition program to update the Poms data parsing program corresponding to the Poms system data;

[0083] Step 602, generating a Req basic file based on UNIX format data obtained by parsing the Poms system data based on the updated Poms data parsing program;

[0084] Step 603: Call the Req file generation program to generate a Req data file according to the Req basic file.

[0085] When generating a Req data file containing the first type of Bloomberg system data, based on the above Table 2, first call the file acquisition program GetPomsFolder.exe to update the Poms data parsing program PomsFileTranslate_Filter.ini corresponding to the Poms system data, which is used to filter the Poms data in the next step of parsing. Then, call FileTranslate.exe to generate \poms\data_fil0 e\YYYYMMDD\mid_file\POMS_ODS_000_BOND_POSITION_FLITER_ALL_YYYYMMDD_00N.dat. Furthermore, based on the UNIX format data obtained by parsing the Poms system data by the updated Poms data parsing program PomsFileTranslate_Filter.ini, call BBReqCheck.exe to obtain the BB_DECIM AL field in the POMS_ODS_000_BOND_POSITION_FLITER_ALL_YYYYMMDD_000N.dat file to generate the Req basic file data.dat. Finally, call the Req file generation program BB_ReqGenarater.exe to generate the Req data file based on the Req basic file data.dat.

[0086] In an embodiment of the present invention, a file acquisition program is called to update a Poms data parsing program corresponding to Poms system data, UNIX format data is obtained by parsing the Poms system data based on the updated Poms data parsing program, a Req basic file is generated, and a Req file generation program is called to generate a Req data file according to the Req basic file, thereby improving the accuracy and efficiency of generating a Req data file containing the first type of Bloomberg system data.

[0087] Figure 7 The implementation flow of the data processing method provided by the seventh embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0088] In one embodiment of the present invention, the source system data includes first-type Kondor system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of the first-type Kondor system data, as Figure 7 As shown, in the data processing method, step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database includes:

[0089] Step 701, obtaining a file to be parsed containing first type Kondor system data from a to-be-parsed file directory of the first type Kondor system;

[0090] Step 702: According to the acquired file to be parsed containing the first type of Kondor system data, a first type of Kondor data parsing program corresponding to the first type of Kondor system data is acquired from a pre-configured parsing program database.

[0091] Step 102, using a parsing program corresponding to the source system data to parse the source system data to obtain target format data, including:

[0092] Step 703: Invoke the first type of Kondor data parsing program corresponding to the first type of Kondor system data through a command line to parse the first type of Kondor system data to obtain UNIX format data.

[0093] Specifically, when the source system data includes the first type Kondor system data (2 batches of market data, T+1, 04:35) and the target format data includes UNIX format data, the data file containing the first type Kondor system data is downloaded to the to-be-parsed file directory Res of the first type Kondor system, and the to-be-parsed file containing the first type Kondor system data is obtained from the to-be-parsed file directory Res of the first type Kondor system, for example, the to-be-parsed file containing the first type Kondor system data is copied to the intermediate directory Mid of the first type Kondor system.

[0094] Then, according to the obtained file to be parsed containing the first type of Kondor system data, the first type of Kondor data parsing program corresponding to the first type of Kondor system data is obtained from the parsing program database, the first type of Kondor data parsing program is called through a command line to parse the first type of Kondor system data, and the parsed file is stored in the first type of Kondor system result directory DEAL. The format of the parsed file is DOS format, and then the parsed DOS format file is converted into target format data, that is, UNIX format data.

[0095] Before calling the first type Kondor data parsing program to parse the first type Kondor system data, it can also include filtering invalid data in the first type Kondor system data, calling the first type Kondor data parsing program to parse the first type Kondor system data, including calling the first type Kondor data parsing program to parse the first type Kondor system data after filtering out the invalid data.

[0096] Table 3 below shows the configuration files and their functions required for parsing the first type of Kondor system data to obtain UNIX format data using the first type of Kondor data parsing program:

[0097]

[0098] Table 3 Configuration files and their functions required to parse the first type of Kondor system data to obtain UNIX format data

[0099] In an embodiment of the present invention, a file to be parsed containing first-type Kondor system data is obtained from a directory of files to be parsed of a first-type Kondor system, and a first-type Kondor data parsing program corresponding to the first-type Kondor system data is obtained from a pre-configured parsing program database according to the obtained file to be parsed containing first-type Kondor system data. The first-type Kondor data parsing program corresponding to the first-type Kondor system data is called through a command line to parse the first-type Kondor system data to obtain UNIX format data, thereby improving the processing efficiency of the first-type Kondor system data.

[0100] Figure 8 The implementation flow of the data processing method provided by the eighth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0101] In one embodiment of the present invention, the source system data includes the second type Kondor system data and / or the third type Kondor system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of the Kondor system data, Figure 8 As shown, in the data processing method, step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database includes:

[0102] Step 801, according to the to-be-parsed file containing the second type of Kondor system data obtained from the to-be-parsed file directory of the second type of Kondor system, obtain the second type of Kondor data parsing program corresponding to the second type of Kondor system data from a pre-configured parsing program database; and / or

[0103] Step 802: According to the to-be-parsed file containing the third type of Kondor system data obtained from the SFTP server, a third type of Kondor data parsing program corresponding to the third type of Kondor system data is obtained from a pre-configured parsing program database;

[0104] Step 102, using a parsing program corresponding to the source system data to parse the source system data to obtain target format data, including:

[0105] Step 803, calling the second type Kondor data parsing program corresponding to the second type Kondor system data through a command line to parse the second type Kondor system data, and obtaining a parsing file corresponding to the second type Kondor system data; and / or

[0106] Step 804, calling the third type Kondor data parsing program corresponding to the third type Kondor system data through a command line to parse the third type Kondor system data, and obtaining a parsing file corresponding to the third type Kondor system data;

[0107] Step 805: convert the parsed file corresponding to the second type Kondor system data and / or the parsed file corresponding to the third type Kondor system data into UNIX format data.

[0108] When the source system data includes the second type Kondor system data (K+2.6 data: 2 batches of T, 20:05, 2 batches of T+1, 05:10) and the target format data includes UNIX format data, based on the above Table 3, the data files containing the first type Kondor system data are downloaded to the to-be-parsed file directory Res of the first type Kondor system, and the to-be-parsed files containing the first type Kondor system data are obtained from the to-be-parsed file directory Res of the first type Kondor system, for example, the to-be-parsed files containing the first type Kondor system data are copied to the intermediate directory Mid of the first type Kondor system.

[0109] Then, according to the obtained file to be parsed containing the first type of Kondor system data, the first type of Kondor data parsing program corresponding to the first type of Kondor system data is obtained from the parsing program database, the first type of Kondor data parsing program is called through a command line to parse the first type of Kondor system data, and the parsed file is stored in the first type of Kondor system result directory DEAL. The format of the parsed file is DOS format, and then the parsed DOS format file is converted into target format data, that is, UNIX format data.

[0110] Before calling the first type Kondor data parsing program to parse the first type Kondor system data, it may also include filtering invalid data in the first type Kondor system data, calling the first type Kondor data parsing program to parse the first type Kondor system data, including calling the first type Kondor data parsing program to parse the first type Kondor system data after filtering out the invalid data, and obtaining a parsing file (K+2.6 parsing file) corresponding to the second type Kondor system data.

[0111] When the source system data includes the third type Kondor system data (K+3.0 data: 2 batches of T, 20:05, 2 batches of T+1, 05:10) and the target format data includes UNIX format data, based on the above Table 3, the data files containing the third type Kondor system data are downloaded to the to-be-parsed file directory Res of the third type Kondor system, and the to-be-parsed files containing the third type Kondor system data are obtained from the to-be-parsed file directory Res of the third type Kondor system, for example, the to-be-parsed files containing the third type Kondor system data are copied to the intermediate directory Mid of the third type Kondor system.

[0112] Then, according to the obtained file to be parsed containing the third type of Kondor system data, the third type of Kondor data parsing program corresponding to the third type of Kondor system data is obtained from the parsing program database, and the third type of Kondor data parsing program is called through the command line to parse the third type of Kondor system data, and the parsed file is stored in the third type of Kondor system result directory DEAL. The format of the parsed file is DOS format, and then the parsed DOS format file is converted into target format data, that is, UNIX format data.

[0113] Among them, before calling the third type Kondor data parsing program to parse the third type Kondor system data, it can also include filtering invalid data in the third type Kondor system data, calling the third type Kondor data parsing program to parse the third type Kondor system data, including calling the third type Kondor data parsing program to parse the third type Kondor system data after filtering out the invalid data, and obtaining the parsing file (K+3.0 parsing file) corresponding to the third type Kondor system data.

[0114] Finally, the parsing file corresponding to the second type of Kondor system data (K+2.6 parsing file) and / or the parsing file corresponding to the third type of Kondor system data (K+3.0 parsing file) are merged and converted into UNIX format data.

[0115] In the embodiment of the present invention, according to the to-be-parsed file containing the second type of Kondor system data obtained from the to-be-parsed file directory of the second type of Kondor system, the second type of Kondor data parsing program corresponding to the second type of Kondor system data is obtained from a pre-configured parsing program database, or according to the to-be-parsed file containing the third type of Kondor system data obtained from the SFTP server, the third type of Kondor data parsing program corresponding to the third type of Kondor system data is obtained from a pre-configured parsing program database; the second type of Kondor system data parsing program corresponding to the second type of Kondor system data is called through a command line to parse the second type of Kondor system data to obtain the parsed file corresponding to the second type of Kondor system data, and / or the third type of Kondor system data parsing program is called through a command line to parse the third type of Kondor system data to obtain the parsed file corresponding to the third type of Kondor system data, and the parsed file corresponding to the second type of Kondor system data and / or the parsed file corresponding to the third type of Kondor system data is converted into UNIX format data, so as to improve the processing efficiency of the Kondor system data.

[0116] Fig. 9 The implementation flow of the data processing method provided by the ninth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0117] In one embodiment of the present invention, the source system data includes Reuter system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Reuter system data, Fig. 9 As shown, in the data processing method, step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database includes:

[0118] Step 901, obtaining a file to be parsed containing Reuter system data from a directory of files to be parsed in the Reuter system;

[0119] Step 902, according to the acquired file to be parsed containing the Reuter system data, a Reuter data parsing program corresponding to the Reuter system data is acquired from a pre-configured parsing program database;

[0120] Step 102, using a parsing program corresponding to the source system data to parse the source system data to obtain target format data, including:

[0121] Step 903: Invoke the Reuter data parsing program corresponding to the Reuter system data through a command line to parse the Reuter system data and obtain UNIX format data.

[0122] Specifically, when the source system data includes Reuter system data (1 batch of market data, T+1, 01:05; 2 batches of market data, T+2, 04:35) and the target format data includes UNIX format data, the data file containing the Reuter system data is downloaded to the to-be-parsed file directory Res of the Reuter system, and the to-be-parsed file containing the Reuter system data is obtained from the to-be-parsed file directory Res of the Reuter system, for example, the to-be-parsed file containing the Reuter system data is copied to the intermediate directory Mid of the Reuter system.

[0123] Then, according to the obtained file to be parsed containing Reuter system data, the Reuter data parsing program corresponding to the Reuter system data is obtained from the parsing program database, and the Reuter data parsing program is called through the command line to parse the Reuter system data, and the parsed file is stored in the Reuter system result directory DEAL. Among them, the format of the parsed file is DOS format, and then the parsed DOS format file is converted into the target format data, that is, UNIX format data.

[0124] Before calling the Reuter data parsing program to parse the Reuter system data, it may also include filtering invalid data in the Reuter system data, calling the Reuter data parsing program to parse the Reuter system data, including calling the Reuter data parsing program to parse the Reuter system data after filtering out the invalid data.

[0125] Table 4 below shows the configuration files and their functions required for parsing Reuter system data to obtain UNIX format data using the Reuter data parsing program:

[0126]

[0127] Table 4. Configuration files and their functions required to parse Reuter system data to obtain UNIX format data

[0128] In an embodiment of the present invention, a file to be parsed containing Reuter system data is obtained from a directory of files to be parsed containing Reuter system data, and based on the obtained file to be parsed containing Reuter system data, a Reuter data parsing program corresponding to the Reuter system data is obtained from a pre-configured parsing program database, and the Reuter system data is parsed by calling the Reuter data parsing program corresponding to the Reuter system data through a command line to obtain UNIX format data, thereby improving the processing efficiency of the Reuter system data.

[0129] Fig.10 The implementation flow of the data processing method provided by the tenth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0130] In one embodiment of the present invention, the source system data includes Wind system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Wind system data, Fig.10 As shown, in the data processing method, step 101, according to the received source system data, obtaining a parsing program corresponding to the source system data from a pre-configured parsing program database includes:

[0131] Step 1001, obtaining a file to be parsed containing Wind system data from a directory of files to be parsed in the Wind system;

[0132] Step 1002, according to the acquired file to be parsed containing Wind system data, a Wind data parsing program corresponding to the Wind system data is acquired from a pre-configured parsing program database;

[0133] Step 102, using a parsing program corresponding to the source system data to parse the source system data to obtain target format data, including:

[0134] Step 1003: Invoke the Wind data parsing program corresponding to the Wind system data through a command line to parse the Wind system data and obtain UNIX format data.

[0135] Specifically, when the source system data includes Wind system data (1 batch of market data, T+1, 01:05; 2 batches of market data, T+2, 04:35) and the target format data includes UNIX format data, the data file containing the Wind system data is downloaded to the to-be-parsed file directory Res of the Wind system, and the to-be-parsed file containing the Wind system data is obtained from the to-be-parsed file directory Res of the Wind system, for example, the to-be-parsed file containing the Wind system data is copied to the intermediate directory Mid of the Wind system.

[0136] Then, according to the obtained file to be parsed containing Wind system data, the Wind data parsing program corresponding to the Wind system data is obtained from the parsing program database, and the Wind data parsing program is called through the command line to parse the Wind system data, and the parsed file is stored in the Wind system result directory DEAL. Among them, the format of the parsed file is DOS format, and then the parsed DOS format file is converted into the target format data, that is, UNIX format data.

[0137] Before calling the Wind data analysis program to analyze the Wind system data, it may also include filtering invalid data in the Wind system data, calling the Wind data analysis program to analyze the Wind system data, including calling the Wind data analysis program to analyze the Wind system data after filtering out the invalid data.

[0138] Table 5 below shows the configuration files and their functions required for parsing Wind system data to obtain UNIX format data using the Wind data parsing program:

[0139]

[0140] Table 5 Configuration files and their functions required to parse Wind system data to obtain UNIX format data

[0141] In an embodiment of the present invention, a file to be parsed containing Wind system data is obtained from a directory of files to be parsed of a Wind system. Based on the obtained file to be parsed containing Wind system data, a Wind data parsing program corresponding to the Wind system data is obtained from a pre-configured parsing program database. The Wind system data is parsed by calling the Wind data parsing program corresponding to the Wind system data through a command line to obtain UNIX format data, thereby improving the processing efficiency of the Wind system data.

[0142] In addition, Table 6 shows the common program files and corresponding function descriptions used in the data processing process of different source systems, and Table 7 shows the common configuration files and corresponding function descriptions used in the data processing process of different source systems:

[0143] SRC DESC FileTranslate.exe Data parsing program FileTranslateK30.exe Data analysis program, dedicated to K+ data processing GetFtpFile.exe Data download program DesCode.exe Password decryption program DesAndReDes.exe Password Encryption Program ConvertUnix.exe Convert files to Unix format FTP_Download_Chk.bat Check if market data is generated

[0144] Table 6 Common program files used in data processing of different source systems and corresponding function descriptions

[0145] INI DESC K3_AddFile.ini K+3.0 parser configuration file for golden data file merging AddFile.ini K+ parser configuration file for golden data file merging FTP_Download_CHK.ini FTP_Download_CHK.bat configuration file, set the log directory KondorFileTranslate.ini K+ parser configuration file MTMFileTranslate.ini K+ parser MTM data class configuration file BloombergFileTranslate.ini Bloomberg Resolver Configuration File PomsFileTranslate.ini Poms resolver configuration file PomsFileTranslate_Filter.ini Poms parser configuration file, used by Bloomberg ReuterFileTranslate.ini Reuters resolver configuration file Site.ini Ftp program configuration file, configure ip, user, password translate.ini Parser configuration files, configuration data, log directories

[0146] Table 7 Common configuration files used in data processing of different source systems and corresponding function descriptions

[0147] The present invention also provides a data processing device, as described in the following embodiments. Since the principles of solving problems by these devices are similar to those of the data processing method, the implementation of these devices can refer to the implementation of the method, and the repeated parts will not be repeated.

[0148] Fig.11 The functional modules of the data processing device provided by the eleventh embodiment of the present invention are shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0149] refer to Fig.11 The modules included in the data processing device are used to execute Figure 1 For details of each step in the corresponding embodiment, please refer to Figure 1 as well as Figure 1 In the embodiment of the present invention, the data processing device includes a parsing program acquisition module 1101 and a data parsing module 1102 .

[0150] The parsing program acquisition module 1101 is used to acquire the parsing program corresponding to the source system data from a pre-configured parsing program database according to the received source system data.

[0151] The data parsing module 1102 is used to parse the source system data using a parsing program corresponding to the source system data to obtain target format data.

[0152] In the embodiment of the present invention, the parsing program acquisition module 1101 acquires the parsing program corresponding to the source system data from the pre-configured parsing program database according to the received source system data, and the data parsing module 1102 parses the source system data using the parsing program corresponding to the source system data to obtain the target format data. The embodiment of the present invention improves the flexibility of processing different source data by pre-configuring the parsing program in the parsing program database, and parses the source system data using the parsing program corresponding to the source system data, thereby obtaining the target format data with a unified format, and improving the efficiency of processing different source data.

[0153] Fig.12 The functional modules of the data processing device provided by the twelfth embodiment of the present invention are shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown, which are described in detail as follows:

[0154] In one embodiment of the present invention, in order to improve the flexibility of the parsing program configuration, reference Fig.12 The modules included in the data processing device are used to execute Figure 2 For details of each step in the corresponding embodiment, please refer to Figure 2 as well as Figure 2The relevant descriptions in the corresponding embodiments are not repeated here. Fig.11 On the basis of the functional modules shown, the data processing device further includes a configuration module 1201 .

[0155] The configuration module 1201 is used to configure the parsing program in the parsing program database according to the received configuration instruction.

[0156] In the embodiment of the present invention, the configuration module 1201 configures the parsing program in the parsing program database according to the received configuration instruction, which can improve the flexibility of the parsing program configuration.

[0157] Fig.13 The structure of the data processing device provided by the thirteenth embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0158] In one embodiment of the present invention, the source system data includes Poms system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Poms system data, refer to Fig.13 The data processing device includes various units for executing Figure 3 For details of each step in the corresponding embodiment, please refer to Figure 3 as well as Figure 3 In the embodiment of the present invention, in the data processing device, the parsing program acquisition module 1101 includes a Poms parsing file acquisition unit 1301 and a Poms parsing program acquisition unit 1302 .

[0159] The Poms parsed file acquisition unit 1301 is used to acquire the to-be-parsed file containing Poms system data from the to-be-parsed file directory of the Poms system.

[0160] The Poms parsing program acquiring unit 1302 is used to acquire a Poms data parsing program corresponding to the Poms system data from a pre-configured parsing program database according to the acquired file to be parsed containing the Poms system data.

[0161] The data parsing module 1102 includes a Poms data parsing unit 1303 .

[0162] The Poms data parsing unit 1303 is used to call the Poms data parsing program corresponding to the Poms system data through a command line to parse the Poms system data and obtain UNIX format data.

[0163] In the embodiment of the present invention, the Poms parsing file obtaining unit 1301 obtains a file to be parsed containing Poms system data from a directory of files to be parsed of the Poms system, the Poms parsing program obtaining unit 1302 obtains a Poms data parsing program corresponding to the Poms system data from a pre-configured parsing program database according to the obtained file to be parsed containing Poms system data, and the Poms data parsing unit 1303 calls the Poms data parsing program corresponding to the Poms system data through a command line to parse the Poms system data and obtain UNIX format data, thereby improving the processing efficiency of the Poms system data.

[0164] Fig.14 The structure of the data processing device provided by the fourteenth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0165] In one embodiment of the present invention, the source system data includes Bloomberg system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Bloomberg system data, refer to Fig.14 The data processing device includes various units for executing Figure 4 For details of each step in the corresponding embodiment, please refer to Figure 4 as well as Figure 4 In the embodiment of the present invention, in the data processing device, the parsing program acquisition module includes a Req file generation unit 1401 and a Bloomberg parsing program acquisition unit 1402 .

[0166] The Req file generating unit 1401 is used to generate a Req data file containing Bloomberg system data.

[0167] The Bloomberg parsing program acquisition unit 1402 is used to acquire a Bloomberg data parsing program corresponding to the Bloomberg system data from a pre-configured parsing program database according to the acquired Req data file containing the Bloomberg system data.

[0168] The data parsing module 1101 includes a Bloomberg data parsing unit 1403 .

[0169] The Bloomberg data parsing unit 1403 is used to call the Bloomberg data parsing program corresponding to the Bloomberg system data through a command line to parse the Bloomberg system data and obtain UNIX format data.

[0170] In the embodiment of the present invention, the Req file generating unit 1401 obtains a file to be parsed containing Bloomberg system data from a directory of files to be parsed of the Bloomberg system, the Bloomberg parsing program obtaining unit 1402 obtains a Bloomberg data parsing program corresponding to the Bloomberg system data from a pre-configured parsing program database according to the obtained file to be parsed containing Bloomberg system data, and the Bloomberg data parsing unit 1403 calls the Bloomberg data parsing program corresponding to the Bloomberg system data through a command line to parse the Bloomberg system data and obtain UNIX format data, thereby improving the processing efficiency of the Bloomberg system data.

[0171] Fig.15 The structure of the Req file generating unit 1401 in the data processing device provided in the fifteenth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0172] In one embodiment of the present invention, the Bloomberg system data includes first type Bloomberg system data and second type Bloomberg system data. In order to improve the accuracy and comprehensiveness of generating the Req data file, refer to Fig.15 The various units included in the Req file generation unit 1401 are used to execute Figure 5 For details of each step in the corresponding embodiment, please refer to Figure 5 as well as Figure 5 In the embodiment of the present invention, the Req file generating unit 1401 includes a first Req file generating unit 1501 or a second Req file generating unit 1502 .

[0173] The first Req file generating unit 1501 is configured to generate a Req data file including first type Bloomberg system data based on UNIX format data obtained by parsing Poms system data with a Poms data parsing program.

[0174] The second Req file generating unit 1502 is configured to generate a Req data file including second type Bloomberg system data.

[0175] In the embodiment of the present invention, the first Req file generating unit 1501 generates a Req data file including the first type of Bloomberg system data based on the UNIX format data obtained by parsing the Poms system data by the Poms data parsing program, and the second Req file generating unit 1502 generates a Req data file including the second type of Bloomberg system data, which can improve the accuracy and comprehensiveness of the generated Req data file.

[0176] Fig.16 The structure of the first Req file generating unit 1501 in the data processing device provided in the sixteenth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0177] In one embodiment of the present invention, in order to improve the accuracy and efficiency of generating the Req data file containing the first type of Bloomberg system data, reference Fig.16 The various sub-units included in the first Req file generating unit 1501 are used to execute Figure 6 For details of each step in the corresponding embodiment, please refer to Figure 6 as well as Figure 6 In the embodiment of the present invention, the first Req file generating unit 1501 includes an updating subunit 1601 , a Req basic file generating subunit 1602 and a first Req file generating subunit 1603 .

[0178] The updating subunit 1601 is used to call the file acquisition program to update the Poms data parsing program corresponding to the Poms system data.

[0179] The Req basic file generating subunit 1602 is used to generate a Req basic file based on the UNIX format data obtained by parsing the Poms system data based on the updated Poms data parsing program.

[0180] The first Req file generating subunit 1603 is used to call the Req file generating program to generate a Req data file according to the Req basic file.

[0181] In the embodiment of the present invention, the updating subunit 1601 calls the file acquisition program to update the Poms data parsing program corresponding to the Poms system data, the Req basic file generating subunit 1602 generates a Req basic file by parsing the UNIX format data obtained by the updated Poms data parsing program, and the first Req file generating subunit 1603 calls the Req file generating program to generate a Req data file according to the Req basic file, which can improve the accuracy and efficiency of generating the Req data file containing the first type of Bloomberg system data.

[0182] Fig.17 The structure of the data processing device provided by the seventeenth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0183] In one embodiment of the present invention, the source system data includes the first type Kondor system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of the first type Kondor system data, refer to Fig.17 The data processing device includes various units for executing Figure 7 For details of each step in the corresponding embodiment, please refer to 7 and Figure 7 In the embodiment of the present invention, in the data processing device, the parsing program acquisition module 1101 includes a first type Kondor parsing file acquisition unit 1701 and a first type Kondor parsing program acquisition unit 1702 .

[0184] The first type Kondor parsed file obtaining unit 1701 is used to obtain the to-be-parsed file containing the first type Kondor system data from the to-be-parsed file directory of the first type Kondor system.

[0185] The first type Kondor parsing program acquiring unit 1702 is configured to acquire the first type Kondor data parsing program corresponding to the first type Kondor system data from a pre-configured parsing program database according to the acquired file to be parsed containing the first type Kondor system data.

[0186] The data parsing module 1102 includes a first type Kondor data parsing unit 1703 .

[0187] The first type Kondor data parsing unit 1703 is used to call the first type Kondor data parsing program corresponding to the first type Kondor system data through a command line to parse the first type Kondor system data to obtain UNIX format data.

[0188] In the embodiment of the present invention, the first-type Kondor parsing file acquiring unit 1701 acquires the to-be-parsed file containing the first-type Kondor system data from the to-be-parsed file directory of the first-type Kondor system; the first-type Kondor parsing program acquiring unit 1702 acquires the first-type Kondor data parsing program corresponding to the first-type Kondor system data from a pre-configured parsing program database according to the acquired to-be-parsed file containing the first-type Kondor system data; the first-type Kondor data parsing unit 1703 calls the first-type Kondor data parsing program corresponding to the first-type Kondor system data through a command line to parse the first-type Kondor system data and acquire UNIX format data, thereby improving the processing efficiency of the first-type Kondor system data.

[0189] Fig.18 The structure of the data processing device provided by the eighteenth embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0190] In one embodiment of the present invention, the source system data includes the second type Kondor system data and / or the third type Kondor system data, and the target format data includes UNIX format data. Fig.18 The data processing device includes various units for executing Figure 8 For details of each step in the corresponding embodiment, please refer to Figure 8 as well as Figure 8 In the embodiment of the present invention, in the data processing device, the parser acquisition module 1101 includes a second type Kondor parser acquisition unit 1801 and / or a third type Kondor parser acquisition unit 1802.

[0191] The second type Kondor parsing program acquiring unit 1801 is used to acquire the second type Kondor data parsing program corresponding to the second type Kondor system data from a pre-configured parsing program database according to the to-be-parsed file containing the second type Kondor system data acquired from the to-be-parsed file directory of the second type Kondor system.

[0192] The third type Kondor parsing program acquiring unit 1802 is configured to acquire a third type Kondor data parsing program corresponding to the third type Kondor system data from a pre-configured parsing program database according to the file to be parsed containing the third type Kondor system data acquired from the SFTP server.

[0193] The data parsing module 1102 includes a second type Kondor data parsing unit 1803 , a third type Kondor data parsing unit 1804 , and a format conversion unit 1805 .

[0194] The second type Kondor data parsing unit 1803 is used to call the second type Kondor data parsing program corresponding to the second type Kondor system data through a command line to parse the second type Kondor system data and obtain a parsing file corresponding to the second type Kondor system data.

[0195] The third type Kondor data parsing unit 1804 is used to call the third type Kondor data parsing program corresponding to the third type Kondor system data through a command line to parse the third type Kondor system data and obtain a parsing file corresponding to the third type Kondor system data.

[0196] The format conversion unit 1805 is used to convert the parsed file corresponding to the second type Kondor system data and / or the parsed file corresponding to the third type Kondor system data into UNIX format data.

[0197] In the embodiment of the present invention, the second-type Kondor parsing program acquiring unit 1801 acquires the second-type Kondor data parsing program corresponding to the second-type Kondor system data from a pre-configured parsing program database according to the to-be-parsed file containing the second-type Kondor system data acquired from the to-be-parsed file directory of the second-type Kondor system, or the third-type Kondor parsing program acquiring unit 1802 acquires the third-type Kondor data parsing program corresponding to the third-type Kondor system data from a pre-configured parsing program database according to the to-be-parsed file containing the third-type Kondor system data acquired from the SFTP server; the second-type Kondor data parsing unit 1803 calls the second-type Kondor data parsing unit 1804 through a command line. The second type Kondor data parsing program corresponding to the type Kondor system data parses the second type Kondor system data to obtain the parsed file corresponding to the second type Kondor system data, and / or the third type Kondor data parsing unit 1804 calls the third type Kondor data parsing program corresponding to the third type Kondor system data through a command line to parse the third type Kondor system data to obtain the parsed file corresponding to the third type Kondor system data, and the format conversion unit 1805 converts the parsed file corresponding to the second type Kondor system data and / or the parsed file corresponding to the third type Kondor system data into UNIX format data, which can improve the processing efficiency of the Kondor system data.

[0198] Fig.19 The structure of the data processing device provided by the embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0199] In one embodiment of the present invention, the source system data includes Reuter system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Reuter system data, reference Fig.19 The data processing device includes various units for executing Fig. 9 For details of each step in the corresponding embodiment, please refer to Fig. 9 as well as Fig. 9 In the embodiment of the present invention, in the data processing device, the parsing program acquisition module 1101 includes a Reuter parsing file acquisition unit 1901 and a Reuter parsing program acquisition unit 1902 .

[0200] The Reuter parsed file acquisition unit 1901 is used to acquire the to-be-parsed file containing the Reuter system data from the to-be-parsed file directory of the Reuter system.

[0201] The Reuter parsing program acquiring unit 1902 is used to acquire the Reuter data parsing program corresponding to the Reuter system data from a pre-configured parsing program database according to the acquired file to be parsed containing the Reuter system data.

[0202] The data analysis module 1102 includes a Reuter data analysis unit 1903 .

[0203] The Reuter data parsing unit 1903 is used to call the Reuter data parsing program corresponding to the Reuter system data through a command line to parse the Reuter system data and obtain UNIX format data.

[0204] In the embodiment of the present invention, the Reuter parsing file acquisition unit 1901 acquires the file to be parsed containing the Reuter system data from the to-be-parsed file directory of the Reuter system, the Reuter parsing program acquisition unit 1902 acquires the Reuter data parsing program corresponding to the Reuter system data from a pre-configured parsing program database based on the acquired file to be parsed containing the Reuter system data, and the Reuter data parsing unit 1903 calls the Reuter data parsing program corresponding to the Reuter system data through a command line to parse the Reuter system data and acquire UNIX format data, thereby improving the processing efficiency of the Reuter system data.

[0205] Fig. 20 The structure of the data processing device provided by the embodiment of the present invention is shown. For the convenience of description, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0206] In one embodiment of the present invention, the source system data includes Wind system data, and the target format data includes UNIX format data. In order to improve the processing efficiency of Wind system data, reference Fig. 20 The data processing device includes various units for executing Fig.10 For details of each step in the corresponding embodiment, please refer to Fig.10 as well as Fig.10 In the embodiment of the present invention, in the data processing device, the parsing program acquisition module 1101 includes a Wind parsing file acquisition unit 2001 and a Wind parsing program acquisition unit 2002 .

[0207] The Wind parsed file acquisition unit 2001 is used to acquire the to-be-parsed file containing Wind system data from the to-be-parsed file directory of the Wind system.

[0208] The Wind parsing program acquisition unit 2002 is used to acquire the Wind data parsing program corresponding to the Wind system data from a pre-configured parsing program database according to the acquired file to be parsed containing the Wind system data.

[0209] The data analysis module 1102 includes a Wind data analysis unit 2003 .

[0210] The Wind data parsing unit 2003 is used to call the Wind data parsing program corresponding to the Wind system data through a command line to parse the Wind system data and obtain UNIX format data.

[0211] In an embodiment of the present invention, a Wind parsing file acquisition unit 2001 acquires a file to be parsed containing Wind system data from a directory of files to be parsed of the Wind system; a Wind parsing program acquisition unit 2002 acquires a Wind data parsing program corresponding to the Wind system data from a pre-configured parsing program database based on the acquired file to be parsed containing Wind system data; a Wind data parsing unit 2003 calls the Wind data parsing program corresponding to the Wind system data through a command line to parse the Wind system data and acquire UNIX format data, thereby improving the processing efficiency of the Wind system data.

[0212] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data processing method when executing the computer program.

[0213] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above data processing method.

[0214] In summary, in the embodiment of the present invention, according to the received source system data, the parsing program corresponding to the source system data is obtained from the pre-configured parsing program database, and the source system data is parsed using the parsing program corresponding to the source system data to obtain target format data. The embodiment of the present invention improves the flexibility of processing different source data by pre-configuring the parsing program in the parsing program database, and at the same time, the source system data is parsed using the parsing program corresponding to the source system data, thereby obtaining target format data with a unified format, and improving the efficiency of processing different source data.

[0215] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0216] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0217] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0219] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: According to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database; The source system data includes Bloomberg system data, the target format data includes UNIX format data, and the Bloomberg system data includes first type Bloomberg system data and second type Bloomberg system data; According to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database, including: calling a file acquisition program to update a Poms data parsing program corresponding to the Poms system data; parsing the Poms system data based on the updated Poms data parsing program to obtain UNIX format data, and generating a Req basic file; Calling a Req file generation program to generate a Req data file according to a Req basic file; or generating a Req data file containing second type Bloomberg system data; acquiring a Bloomberg data parsing program corresponding to the Bloomberg system data from a pre-configured parsing program database according to the acquired Req data file containing the Bloomberg system data; The Bloomberg data parsing program corresponding to the Bloomberg system data is called through the command line to parse the Bloomberg system data and obtain the UNIX format data.

2. The data processing method according to claim 1, characterized in that: Also includes: The resolvers in the resolver database are configured according to the received configuration instructions.

3. The data processing method according to claim 1, characterized in that: The source system data includes Poms system data, and the target format data includes UNIX format data. According to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database, including: Get the file to be parsed containing Poms system data from the Poms system's file to be parsed directory; According to the acquired file to be parsed containing the Poms system data, a Poms data parsing program corresponding to the Poms system data is acquired from a pre-configured parsing program database; Use the parsing program corresponding to the source system data to parse the source system data and obtain the target format data, including: The Poms data parsing program corresponding to the Poms system data is called through the command line to parse the Poms system data to obtain UNIX format data.

4. The data processing method according to claim 1, characterized in that: The source system data includes first type Kondor system data, the target format data includes UNIX format data, and according to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database, including: Obtaining a file to be parsed containing first-type Kondor system data from a directory of files to be parsed of the first-type Kondor system; According to the acquired file to be parsed containing the first type of Kondor system data, a first type of Kondor data parsing program corresponding to the first type of Kondor system data is acquired from a pre-configured parsing program database; Use the parsing program corresponding to the source system data to parse the source system data and obtain the target format data, including: The first type of Kondor system data is parsed by calling a first type of Kondor data parsing program corresponding to the first type of Kondor system data through a command line to obtain UNIX format data.

5. The data processing method according to claim 1, characterized in that: The source system data includes the second type Kondor system data and / or the third type Kondor system data, the target format data includes UNIX format data, and according to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database, including: According to the to-be-parsed file containing the second type of Kondor system data obtained from the to-be-parsed file directory of the second type of Kondor system, obtaining the second type of Kondor data parsing program corresponding to the second type of Kondor system data from a pre-configured parsing program database; and / or According to the to-be-parsed file containing the third type of Kondor system data obtained from the SFTP server, a third type of Kondor data parsing program corresponding to the third type of Kondor system data is obtained from a pre-configured parsing program database; Use the parsing program corresponding to the source system data to parse the source system data and obtain the target format data, including: Invoke the second type of Kondor data parsing program corresponding to the second type of Kondor system data through a command line to parse the second type of Kondor system data, and obtain a parsing file corresponding to the second type of Kondor system data; and / or Invoke the third type of Kondor data parsing program corresponding to the third type of Kondor system data through the command line to parse the third type of Kondor system data, and obtain the parsing file corresponding to the third type of Kondor system data; The parsed file corresponding to the second type Kondor system data and / or the parsed file corresponding to the third type Kondor system data is converted into UNIX format data.

6. The data processing method according to claim 1, characterized in that: The source system data includes Reuter system data, and the target format data includes UNIX format data. According to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database, including: Obtain the file to be parsed containing the Reuter system data from the Reuter system's directory of files to be parsed; According to the acquired file to be parsed containing the Reuter system data, a Reuter data parsing program corresponding to the Reuter system data is acquired from a pre-configured parsing program database; Use the parsing program corresponding to the source system data to parse the source system data and obtain the target format data, including: The Reuter data parsing program corresponding to the Reuter system data is called through the command line to parse the Reuter system data and obtain UNIX format data.

7. The data processing method according to claim 1, characterized in that: The source system data includes Wind system data, and the target format data includes UNIX format data. According to the received source system data, a parsing program corresponding to the source system data is obtained from a pre-configured parsing program database, including: Obtain the file to be parsed containing Wind system data from the file directory to be parsed of the Wind system; According to the acquired file to be parsed containing Wind system data, a Wind data parsing program corresponding to the Wind system data is acquired from a pre-configured parsing program database; Use the parsing program corresponding to the source system data to parse the source system data and obtain the target format data, including: The Wind data parsing program corresponding to the Wind system data is called through the command line to parse the Wind system data and obtain UNIX format data.

8. A data processing device, characterized in that: include: A parsing program acquisition module, used to acquire a parsing program corresponding to the source system data from a pre-configured parsing program database according to the received source system data; A data parsing module is used to parse the source system data using a parsing program corresponding to the source system data to obtain target format data; The source system data includes Bloomberg system data, the target format data includes UNIX format data, and the parsing program acquisition modules include: A Req file generating unit, used for generating a Req data file containing Bloomberg system data; A Bloomberg parsing program acquisition unit, configured to acquire a Bloomberg data parsing program corresponding to the Bloomberg system data from a pre-configured parsing program database according to the acquired Req data file containing the Bloomberg system data; The data analysis module includes: A Bloomberg data parsing unit, used to parse the Bloomberg system data by calling a Bloomberg data parsing program corresponding to the Bloomberg system data through a command line to obtain UNIX format data; The Bloomberg system data includes first-type Bloomberg system data and second-type Bloomberg system data, and the Req file generating unit includes: A first Req file generating unit is configured to generate a Req data file including first type Bloomberg system data based on UNIX format data obtained by parsing Poms system data with a Poms data parsing program; or A second Req file generating unit, configured to generate a Req data file containing second type Bloomberg system data; The first Req file generating unit includes: The update subunit is used to call the file acquisition program to update the Poms data parsing program corresponding to the Poms system data; A Req basic file generation subunit is used to generate a Req basic file based on UNIX format data obtained by parsing Poms system data based on an updated Poms data parsing program; The first Req file generating subunit is used to call the Req file generating program to generate a Req data file according to the Req basic file.

9. The data processing device according to claim 8, characterized in that Also includes: The configuration module is used to configure the parsing program in the parsing program database according to the received configuration instructions.

10. The data processing device according to claim 8, characterized in that The source system data includes Poms system data, the target format data includes UNIX format data, and the parsing program acquisition modules include: A Poms parsed file acquisition unit, used to acquire a file to be parsed containing Poms system data from a directory of files to be parsed in the Poms system; A Poms parsing program acquisition unit, configured to acquire a Poms data parsing program corresponding to the Poms system data from a pre-configured parsing program database according to the acquired file to be parsed containing the Poms system data; The data analysis module includes: The Poms data parsing unit is used to call the Poms data parsing program corresponding to the Poms system data through a command line to parse the Poms system data and obtain UNIX format data.

11. The data processing device according to claim 8, characterized in that The source system data includes the first type of Kondor system data, the target format data includes UNIX format data, and the parsing program acquisition module includes: A first-type Kondor parsed file acquisition unit, configured to acquire a to-be-parsed file containing first-type Kondor system data from a to-be-parsed file directory of the first-type Kondor system; a first-type Kondor parsing program acquiring unit, configured to acquire, according to the acquired file to be parsed containing the first-type Kondor system data, a first-type Kondor data parsing program corresponding to the first-type Kondor system data from a pre-configured parsing program database; The data analysis module includes: The first type Kondor data parsing unit is used to call the first type Kondor data parsing program corresponding to the first type Kondor system data through a command line to parse the first type Kondor system data and obtain UNIX format data.

12. The data processing device according to claim 8, characterized in that The source system data includes the second type Kondor system data and / or the third type Kondor system data, the target format data includes UNIX format data, and the parsing program acquisition module includes: a second-type Kondor parsing program acquiring unit, configured to acquire a second-type Kondor data parsing program corresponding to the second-type Kondor system data from a pre-configured parsing program database according to a file to be parsed containing the second-type Kondor system data acquired from a directory of files to be parsed of the second-type Kondor system; and / or a third-type Kondor parsing program acquiring unit, configured to acquire a third-type Kondor data parsing program corresponding to the third-type Kondor system data from a pre-configured parsing program database according to a file to be parsed containing the third-type Kondor system data acquired from the SFTP server; The data analysis module includes: The second type Kondor data parsing unit is used to call the second type Kondor data parsing program corresponding to the second type Kondor system data through a command line to parse the second type Kondor system data and obtain a parsing file corresponding to the second type Kondor system data; and / or A third type Kondor data parsing unit, configured to call a third type Kondor data parsing program corresponding to the third type Kondor system data through a command line to parse the third type Kondor system data, and obtain a parsing file corresponding to the third type Kondor system data; The format conversion unit is used to convert the parsed file corresponding to the second type Kondor system data and / or the parsed file corresponding to the third type Kondor system data into UNIX format data.

13. The data processing device according to claim 8, characterized in that The source system data includes Reuter system data, the target format data includes UNIX format data, and the parsing program acquisition module includes: A Reuter parsed file acquisition unit, used for acquiring a to-be-parsed file containing Reuter system data from a to-be-parsed file directory of the Reuter system; A Reuter parsing program acquisition unit, configured to acquire a Reuter data parsing program corresponding to the Reuter system data from a pre-configured parsing program database according to the acquired file to be parsed containing the Reuter system data; The data analysis module includes: The Reuter data parsing unit is used to call the Reuter data parsing program corresponding to the Reuter system data through a command line to parse the Reuter system data and obtain UNIX format data.

14. The data processing device according to claim 8, characterized in that The source system data includes Wind system data, the target format data includes UNIX format data, and the parsing program acquisition modules include: A Wind parsed file acquisition unit, used for acquiring a file to be parsed containing Wind system data from a file directory to be parsed of the Wind system; A Wind parsing program acquisition unit, configured to acquire a Wind data parsing program corresponding to the Wind system data from a pre-configured parsing program database according to the acquired file to be parsed containing the Wind system data; The data analysis module includes: The Wind data parsing unit is used to parse the Wind system data by calling the Wind data parsing program corresponding to the Wind system data through a command line to obtain UNIX format data.

15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 7 is implemented.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the data processing method according to any one of claims 1 to 7.

Images