A data processing method and apparatus

Abstract

The application discloses a data processing method which can be applied to the financial field or other fields. The method comprises the following steps: receiving a first file, judging whether the first file is an online file; adding identification information to the first file in the case of determining that the first file is an online file; further determining the total number of files including the identification information; and writing the files including the identification information into a data lake in the case that the total number is equal to the total number of online files issued by an upstream node, wherein the total number of online files is determined according to the online file list issued by the upstream node. Since the total number is equal to the total number of online files issued by the upstream node, it indicates that all online files have been received. According to the method, the online files are written into the data lake only in the case that all online files have been received, so that the I / O resources are not wasted due to the writing of the online files with missing data into the data lake.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular, to a data processing method and device. BACKGROUND

[0002] A data lake can receive a large amount of business data issued by an upstream every day, and the business data can be used to schedule a workflow.

[0003] At present, the business data issued by the upstream can be an online file, and when the number of online files is large, writing a large number of online files into the data lake consumes a large amount of input / output (IO) resources. If the online file is missing, the workflow cannot be normally scheduled using the online file, and in this case, the IO resources consumed for writing the missing online file are essentially wasted.

[0004] Therefore, there is an urgent need for a solution to solve the above problems. SUMMARY

[0005] To solve the above technical problems, the embodiments of the present application provide a data processing method and device.

[0006] In a first aspect, the embodiments of the present application provide a data processing method, which comprises:

[0007] receiving a first file;

[0008] determining whether the first file is an online file;

[0009] in a case where it is determined that the first file is an online file, adding identification information to the first file;

[0010] determining a total number of files including the identification information, wherein the files including the identification information include the first file;

[0011] in a case where the total number is equal to a total number of online files issued by an upstream node, writing the files including the identification information into a data lake, wherein the total number of online files is determined according to an online file list issued by the upstream node.

[0012] Optionally, the adding identification information to the first file comprises:

[0013] adding suffix information to the first file.

[0014] Optionally, the method further comprises:

[0015] in a case where it is determined that the first file is not an online file, writing the first file into the data lake and adding a record corresponding to the first file in a file upload list;

[0016] determining whether files in a table to which the first file belongs have all been written into the data lake;

[0017] if it is determined that the files in the table to which the first file belongs have all been written into the data lake, adding a record of the table to which the first file belongs in a file completion list.

[0018] Optionally, the determining whether the files in the table to which the first file belongs have all been written into the data lake comprises:

[0019] comparing the file upload list and a file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake, the file configuration table comprising file lists of a plurality of tables, the plurality of tables including the table to which the first file belongs.

[0020] Optionally, the comparing the file upload list and the file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake comprises:

[0021] extracting index information of the first file, the index information being used to indicate the table to which the first file belongs;

[0022] determining at least one record with the index information from the file upload list based on the index information;

[0023] determining a file list of the table to which the first file belongs from the file configuration table based on the index information;

[0024] comparing the at least one record and the file list of the table to which the first file belongs to determine whether the files in the table to which the first file belongs have all been written into the data lake.

[0025] In a second aspect, an embodiment of the present application provides a data processing apparatus, the apparatus comprising:

[0026] a receiving unit configured to receive a first file;

[0027] a judging unit configured to judge whether the first file is an online file;

[0028] a first adding unit configured to add identification information for the first file in a case where it is determined that the first file is an online file;

[0029] The first determining unit is configured to determine a total number of files including the identification information, the files including the identification information including the first file.

[0030] The first writing unit is configured to write the files including the identification information into the data lake in a case where the total number is equal to a total number of online files issued by an upstream node, wherein the total number of online files is determined according to an online file list issued by the upstream node.

[0031] Optionally, the first adding unit is configured to add suffix information to the first file.

[0032] Optionally, the apparatus further includes:

[0033] The second writing unit is configured to write the first file into the data lake and add a record corresponding to the first file in a file upload list in a case where it is determined that the first file is not an online file.

[0034] The second determining unit is configured to determine whether files in a table to which the first file belongs have all been written into the data lake.

[0035] The second adding unit is configured to add a record of the table to which the first file belongs in a file completion list if it is determined that the files in the table to which the first file belongs have all been written into the data lake.

[0036] Optionally, the second determining unit is configured to:

[0037] Compare the file upload list and a file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake, the file configuration table including file lists of a plurality of tables, the plurality of tables including the table to which the first file belongs.

[0038] Optionally, the comparing the file upload list and the file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake includes:

[0039] Extracting index information of the first file, the index information being used to indicate the table to which the first file belongs.

[0040] Determining at least one record having the index information from the file upload list based on the index information.

[0041] Determining a file list of the table to which the first file belongs from the file configuration table based on the index information.

[0042] comparing the at least one record and a file list of the table to which the first file belongs, to determine whether files in the table to which the first file belongs have all been written into the data lake.

[0043] Compared with the prior art, the embodiments of the present application have the following advantages:

[0044] The embodiments of the present application provide a data processing method, in one example, the method comprises: receiving a first file, and determining whether the first file is an online file. In the case of determining that the first file is an online file, adding identification information to the first file; further, determining a total number of files including the identification information, the files including the identification information including the first file; in the case of the total number being equal to a total number of online files issued by an upstream node, writing the files including the identification information into a data lake, wherein the total number of online files is determined according to an online file list issued by the upstream node. Since the total number is equal to the total number of online files issued by the upstream node, it means that all online files have been received. As can be seen, using the present solution, in the case of determining that all online files have been received, the online files are written into the data lake, thereby avoiding wasting I / O resources by writing online files with missing data into the data lake. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0046] Figure 1 A flowchart of a data processing method provided by the embodiments of the present application;

[0047] Figure 2 A flowchart of another data processing method provided by the embodiments of the present application;

[0048] Figure 3 A structural diagram of a data processing device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0049] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, and obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0050] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, and obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0051] Exemplary method

[0052] Referring to Figure 1 , which is a flowchart of a data processing method provided by an embodiment of the present application. In the embodiment, the method can be performed by a server, and the method can include the following steps: S101-S105.

[0053] S101: receiving a first file.

[0054] In an example, the first file can be a file sent by an upstream node to the server.

[0055] S102: determining whether the first file is an online file.

[0056] In the embodiment of the present application, the online file refers to a sub-file obtained by splitting and compressing a business data file when the business data file is large.

[0057] In an example, whether the first file is an online file can be determined according to the format of the first file.

[0058] S103: adding identification information to the first file in a case where it is determined that the first file is an online file.

[0059] S104: determining a total number of files including the identification information, the files including the identification information including the first file.

[0060] In the embodiment of the present application, after it is determined that the first file is an online file, identification information can be added to the first file. In an example, a pre-processing program can be used to add the identification information to the first file.

[0061] The embodiment of the present application does not specifically limit the identification information, and the identification information can be, for example, suffix information, for example, adding suffix information to the file name of the first file.

[0062] After adding the identification information to the first file, a total number of files including the identification information can be determined, i.e., a total number of online files belonging to the same service file as the first file and having been received.

[0063] S105: In a case where the total number is equal to a total number of online files issued by an upstream node, files including the identification information are written into the data lake, wherein the total number of online files is determined according to an online file list issued by the upstream node.

[0064] In one example, the upstream node can send an online file list to the server, which may, for example, include information of online files included in a service data file. Based on the online file list, a total number of online files included in the service data file can be determined.

[0065] If the total number is equal to a total number of online files issued by an upstream node, the upstream node has sent all online files included in a service data file to the server. In this case, the server can write files including the identification information (i.e., all online files corresponding to the service data file) into the data lake. As can be seen, according to the present solution, online files are written into the data lake only in a case where all online files have been received, thereby avoiding wasting I / O resources caused by writing online files with missing data into the data lake.

[0066] In yet another example, if the first file is not an online file, the server can perform Figure 2 the method shown in FIG. 2, Figure 2 a flowchart of another data processing method provided by an embodiment of the present application. Figure 2 The method shown in FIG. 2 may, for example, include the following S201-S203.

[0067] S201: The first file is written into the data lake, and a record corresponding to the first file is added to a file upload list.

[0068] For a non-online file, after receiving the first file, the server can write the first file into the data lake and add a record corresponding to the first file to a file upload list to record that the first file has been written into the data lake. The file upload list is used to record information of files that have been written into the data lake. The record corresponding to the first file may, for example, include identification information of the first file, which may, for example, include a system to which the first file belongs, a name of a table to which the first file belongs, and a region number corresponding to the table to which the first file belongs, etc.

[0069] S202: Determine whether files in a table to which the first file belongs have all been written into the data lake.

[0070] In one example, after the first file is written into the data lake, the server can further determine whether files in a table to which the first file belongs have all been written into the data lake.

[0071] In yet another example, the server can query a file completion record including a table to which the first file belongs, to determine whether files in the table to which the first file belongs have all been written into the data lake.

[0072] In one example, the server can compare the file upload list and a file configuration table to determine whether files in a table to which the first file belongs have all been written into the data lake, the file configuration table including file lists of a plurality of tables, the plurality of tables including the table to which the first file belongs. For ease of description, the table to which the first file belongs is referred to as table A, and the file configuration table includes a file list of the table A, which may, for example, include identifiers of each file included in the table A.

[0073] In one example, "the server can compare the file upload list and a file configuration table to determine whether files in a table to which the first file belongs have all been written into the data lake" can be implemented through steps A1-A4 as follows in a specific implementation.

[0074] A1: Extract index information of the first file, the index information being used to indicate a table to which the first file belongs.

[0075] The index information of the first file mentioned this time may, for example, be the name of the system to which the first file belongs, the name of the table to which the first file belongs, and the area number corresponding to the table to which the first file belongs. In the embodiments of the present application, the first file can be analyzed, for example, the name of the first file or the content of the first file is analyzed to obtain the index information of the first file.

[0076] A2: Based on the index information, at least one record with the index information is determined from the file upload list.

[0077] The at least one record with the index information is obtained by searching in the file upload list with the index information as the index. It is not difficult to understand that the at least one record corresponds to files in the table A that have been written into the data lake. For example, if three records are found, it means that three files in the table A have been written into the data lake.

[0078] A3: Based on the index information, determine the file list of the table to which the first file belongs from the file configuration table.

[0079] A4: Compare the at least one record with the file list of the table to which the first file belongs to determine whether all the files in the table to which the first file belongs have been written to the data lake.

[0080] Using the index information as an index, the file configuration table can be searched to obtain the file list corresponding to table A.

[0081] In one example, based on the file list corresponding to Table A, the number of files included in Table A can be determined. Therefore, by comparing the number of the at least one record with the number of files included in Table A, it can be determined whether all files in the table to which the first file belongs have been written to the data lake.

[0082] In another example, the file list in table A may include, for example, the identifiers of each file included in table A. Therefore, by comparing the information in the at least one record with the identifiers of each file included in table A, it can be determined whether all files in the table to which the first file belongs have been written to the data lake.

[0083] S203: If all files in the table to which the first file belongs have been written to the data lake, then add a record of the table to which the first file belongs to the file arrival list.

[0084] If, after executing S202, it is determined that all files in the table to which the first file belongs have been written to the data lake, then a record of the table to which the first file belongs can be added to the file arrival list. The file arrival list can record information about the tables to which all files have been written to the data lake.

[0085] Specifically, a record of the table to which the first file belongs is added to the file arrival list. For example, index information of the aforementioned identifier table A can be added. For instance, the system to which table A belongs, the name of table A, and the area number corresponding to table A are written into the file arrival list.

[0086] use Figure 2 The method shown allows the server to first query the file arrival list when it needs to use table A for workflow scheduling. If the file arrival list includes information about table A, then the operation of using table A for workflow scheduling is executed. This avoids invalidating some of the aforementioned operations by directly using table A for workflow scheduling if not all files in table A have been written to the data lake.

[0087] Exemplary apparatus

[0088] Based on the method provided in the above embodiments, the embodiments of the present application further provide an apparatus, which will be described below with reference to the accompanying drawings.

[0089] Referring to Figure 3 The figure is a structural schematic diagram of a data processing apparatus provided by the embodiments of the present application. The apparatus 300 may, for example, specifically include: a receiving unit 301, a judging unit 302, a first adding unit 303, a first determining unit 304, and a first writing unit 305.

[0090] The receiving unit 301 is configured to receive a first file.

[0091] The judging unit 302 is configured to judge whether the first file is an online file.

[0092] The first adding unit 303 is configured to add identification information to the first file in a case where it is determined that the first file is an online file.

[0093] The first determining unit 304 is configured to determine a total number of files including the identification information, the files including the identification information including the first file.

[0094] The first writing unit 305 is configured to write the files including the identification information into a data lake in a case where the total number is equal to a total number of online files issued by an upstream node, wherein the total number of online files is determined according to an online file list issued by the upstream node.

[0095] Optionally, the first adding unit 303 is configured to add suffix information to the first file.

[0096] Optionally, the apparatus further includes:

[0097] The second writing unit is configured to write the first file into the data lake and add a record corresponding to the first file in a file upload list in a case where it is determined that the first file is not an online file.

[0098] The second determining unit is configured to determine whether files in a table to which the first file belongs have all been written into the data lake.

[0099] The second adding unit is configured to add a record of the table to which the first file belongs in a file completion list if it is determined that the files in the table to which the first file belongs have all been written into the data lake.

[0100] Optionally, the second determining unit is configured to:

[0101] Comparing the file upload list and a file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake, the file configuration table including file lists of a plurality of tables, the plurality of tables including the table to which the first file belongs.

[0102] Optionally, the comparing the file upload list and the file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake includes:

[0103] Extracting index information of the first file, the index information being used to indicate the table to which the first file belongs;

[0104] Determining at least one record with the index information from the file upload list based on the index information;

[0105] Determining a file list of the table to which the first file belongs from the file configuration table based on the index information;

[0106] Comparing the at least one record and the file list of the table to which the first file belongs to determine whether the files in the table to which the first file belongs have all been written into the data lake.

[0107] Since the apparatus 300 is the apparatus corresponding to the method provided in the above method embodiments, the specific implementation of each unit of the apparatus 300 is the same as the same concept as the above method embodiments, and therefore, the specific implementation of each unit of the apparatus 300 can be referred to the description part of the above method embodiments, which will not be described here again.

[0108] It should be noted that the data processing method and apparatus provided by the present application can be used in the financial field or other fields. For example, it can be applied to the pre-processing stage of workflow scheduling in the financial field. Other fields are any field other than the financial field, for example, the data storage field. The above is only an example and does not limit the application field of the data processing method and apparatus provided by the present application.

[0109] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0110] It is to be understood that the application is not limited to the precise construction herein described and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope thereof. The scope of the application is limited only by the claims that follow.

[0111] The above description is merely the preferred embodiment of this application, and is not intended to limit the application. Any modification, equivalent replacement and improvement made without departing from the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A data processing method, characterized by, The method comprises: receiving a first file; determining whether the first file is an online file; adding identification information to the first file if it is determined that the first file is an online file; determining a total number of files including the identification information, the files including the identification information including the first file; writing the files including the identification information to a data lake if the total number is equal to a total number of online files issued by an upstream node, wherein the total number of online files is determined according to an online file list issued by the upstream node; The method further comprises: if it is determined that the first file is not an online file, writing the first file to the data lake and adding a record corresponding to the first file in a file upload list; determining whether files in a table to which the first file belongs have all been written to the data lake; if the files in the table to which the first file belongs have all been written to the data lake, adding a record of the table to which the first file belongs in a file completion list; wherein: The determination of whether the files in the table to which the first file belongs have all been written to the data lake comprises: comparing the file upload list and a file configuration table to determine whether the files in the table to which the first file belongs have all been written to the data lake, the file configuration table including file lists of a plurality of tables, the plurality of tables including the table to which the first file belongs; The comparison of the file upload list and the file configuration table to determine whether the files in the table to which the first file belong have all been written to the data lake comprises: extracting index information of the first file, the index information being used to indicate the table to which the first file belongs; determining at least one record having the index information from the file upload list based on the index information; determining a file list of the table to which the first file belongs from the file configuration table based on the index information; comparing the at least one record and the file list of the table to which the first file belongs to determine whether the files in the table to which the first file belong have all been written to the data lake.

2. The method of claim 1, wherein, The adding of the identification information to the first file comprises: adding suffix information to the first file.

3. A data processing apparatus, characterized by, The apparatus comprises: a receiving unit configured to receive a first file; a determining unit configured to determine whether the first file is an online file; a first adding unit configured to add identification information to the first file if it is determined that the first file is an online file; a first determining unit configured to determine a total number of files including the identification information, the files including the identification information including the first file; a first writing unit configured to write the files including the identification information to a data lake if the total number is equal to a total number of online files issued by an upstream node, wherein the total number of online files is determined according to an online file list issued by the upstream node; The apparatus further comprises: The second writing unit is configured to write the first file into the data lake and add a record corresponding to the first file in a file upload list if it is determined that the first file is not an online file. The second determining unit is configured to determine whether files in a table to which the first file belongs have all been written into the data lake. The second adding unit is configured to add a record of the table to which the first file belongs in a file completion list if it is determined that the files in the table to which the first file belongs have all been written into the data lake. The second determining unit is specifically configured to: Compare the file upload list and a file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake, the file configuration table including file lists of a plurality of tables, and the plurality of tables including the table to which the first file belongs. The comparison of the file upload list and the file configuration table to determine whether the files in the table to which the first file belongs have all been written into the data lake includes: Extracting index information of the first file, the index information being used to indicate the table to which the first file belongs; Determining at least one record with the index information from the file upload list based on the index information; Determining a file list of the table to which the first file belongs from the file configuration table based on the index information; Comparing the at least one record and the file list of the table to which the first file belongs to determine whether the files in the table to which the first file belongs have all been written into the data lake.

4. The apparatus of claim 3, wherein, The first adding unit is configured to add suffix information to the first file.

Images