Data processing method and device, electronic equipment and program product

Abstract

The invention provides a data processing method which comprises the following steps: receiving a to-be-uploaded data file, and extracting a header field of the to-be-uploaded data file; under the condition of determining that the configuration file matched with the header field exists, extracting a data field from the to-be-uploaded data file according to the configuration file, filling the data field into an interface uploading document, and taking the filled interface uploading document as a target uploading document; when it is determined that the configuration file matched with the header field does not exist, establishing a configuration file for the header field, extracting the data field from the to-be-uploaded data file through the established configuration file, filling the data field into an interface uploading document, and taking the filled interface uploading document as a target uploading document; and uploading the target uploading document to the target data platform through the data uploading interface. The problem that uploading interfaces need to be developed for multiple times due to incompatibility of header fields is solved, and the data maintenance cost is reduced.

Description

Technical Field

[0001] The present disclosure belongs to the field of Internet technology, and in particular relates to a data processing method and device, an electronic device, and a program product. Background Art

[0002] In the field of data processing, received data files often need to be adaptively configured so that the data processing platform can clean and upload the data files.

[0003] However, the fields of data files from different sources are often different. In order to ensure the upload of data file contents from different sources in multiple scenarios, when the fields of data files change, developers are required to redevelop and launch the upload interface, which leads to excessive data maintenance costs.

[0004] To address the above problems, no effective solution has been proposed yet.

[0005] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention

[0006] The present disclosure aims to provide a data processing method, an electronic device and a program product, which can realize compatible uploading of data of different users in different scenarios, so as to reduce the workload of development and reduce the cost of data maintenance.

[0007] In a first aspect, the present disclosure provides a data processing method, which may include:

[0008] Receiving a data file to be uploaded, and extracting a header field of the data file to be uploaded;

[0009] If it is determined that there is a configuration file matching the header field, extract the data field from the data file to be uploaded according to the configuration file and fill it into the interface upload document, and the filled interface upload document is the target upload document;

[0010] In the case where it is determined that there is no configuration file matching the header field, a configuration file is created for the header field, and data fields are extracted from the data file to be uploaded through the created configuration file and filled into the interface upload document, and the filled interface upload document is used as the target upload document;

[0011] The target upload document is uploaded to the target data platform through the data upload interface.

[0012] Optionally, the establishing a configuration file for the header field includes:

[0013] Acquire a transmission requirement uploaded by an operation node for the data file to be uploaded, wherein the file transmission requirement carries a data field identifier set that needs to be uploaded for the data file to be uploaded;

[0014] Obtaining the header fields of the data file to be uploaded;

[0015] Searching the identifiers of the data fields in the data field set in the header field of the data file to be uploaded to determine the data position sequence of the data fields in the data field set in the data file to be uploaded;

[0016] A configuration file matching the header field of the data file to be uploaded is established according to the data field identification set and the determined data position sequence.

[0017] Optionally, after obtaining the transmission requirement for the data file to be uploaded uploaded by the operation node, the method further includes:

[0018] Retrieve an interface upload document, wherein the interface upload document includes a plurality of data statistics items, wherein each data statistics item includes: a data field identifier, a data type of the data field, an upper limit of a data length of the data field, and data content represented by the data field;

[0019] Traversing the data field identification set to determine whether each data field identification in the data field identification set has been recorded in the interface upload document;

[0020] In the case where it is determined that there are data field identifiers that are not recorded in the interface upload document, a corresponding data statistic item is added to the interface upload document for each unrecorded data field identifier.

[0021] Optionally, the adding of corresponding data statistics items in the interface upload document for each unrecorded field identifier includes:

[0022] In the reserved field position in the interface upload document, for each unrecorded field identifier, a corresponding data statistics item is added in the interface upload document.

[0023] Optionally, the data file to be uploaded carries a user identifier;

[0024] Determine whether there is a configuration file matching the header field, including:

[0025] Obtaining a user ID from the data file to be uploaded;

[0026] Obtaining one or more configuration files associated with the user identifier;

[0027] The data field identifiers in the one or more configuration files are matched with those in the header field to determine whether there is a configuration file adapted to the header field.

[0028] Optionally, the method further comprises:

[0029] The original interface upload document is updated through the target service application, and the original interface upload document is replaced with the interface upload document to which the corresponding data statistics item is added, wherein the target service application is a service application that provides consistency service for distributed applications.

[0030] In a second aspect, a data processing apparatus is provided, which may include:

[0031] A receiving module, configured to receive a data file to be uploaded and extract a header field of the data file to be uploaded;

[0032] A filling module is configured to extract data fields from the data file to be uploaded according to the configuration file and fill them into the interface upload document when it is determined that there is a configuration file matching the header field, and the interface upload document that has been filled is the target upload document;

[0033] an establishing module, configured to establish a configuration file for the header field if it is determined that there is no configuration file matching the header field, and extract data fields from the data file to be uploaded through the established configuration file and fill them into the interface upload document, and the filled interface upload document is the target upload document;

[0034] The upload module is configured to upload the target upload document to the target data platform through the data upload interface.

[0035] Optionally, the establishing module includes:

[0036] A first acquisition unit is configured to acquire a transmission requirement for the data file to be uploaded uploaded by the operation node, wherein the file transmission requirement carries a data field identifier set that needs to be uploaded for the data file to be uploaded;

[0037] A second acquisition unit is configured to acquire a header field of the data file to be uploaded;

[0038] A search unit configured to search for each data field identifier in the data field set in the header field of the data file to be uploaded, so as to determine the data position sequence of each data field in the data field set in the data file to be uploaded;

[0039] The generating unit is configured to establish a configuration file matching the header field of the data file to be uploaded according to the data field identification set and the determined data position sequence.

[0040] In a third aspect, an electronic device is provided, comprising a processor and a memory for storing instructions executable by the processor, wherein the processor implements the steps of the method described in the first aspect when executing the instructions.

[0041] In a fourth aspect, a program product is provided, comprising a computer program, wherein the computer program implements the steps of the method described in the first aspect when executed by a processor.

[0042] The disclosed embodiment discloses a data processing method, which can receive a data file to be uploaded and extract the header field of the data file to be uploaded; when it is determined that there is a configuration file matching the header field, extract the data field from the data file to be uploaded according to the configuration file and fill it into the interface upload document, and the filled interface upload document is the target upload document; when it is determined that there is no configuration file matching the header field, establish a configuration file for the header field, and extract the data field from the data file to be uploaded through the established configuration file and fill it into the interface upload document, and the filled interface upload document is the target upload document; upload the target upload document to the target data platform through the data upload interface. The method of the disclosed embodiment solves the technical problem that the existing header field is incompatible, which requires multiple development of upload interfaces, and the data maintenance cost is too high, and achieves the compatible upload of data for different users and different scenarios, as well as the technical effect of reducing the workload of development and reducing the data maintenance cost.

[0043] Other optional features and technical effects of the embodiments of the present disclosure are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The elements shown are not limited by the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:

[0045] Figure 1 One of the flowcharts of the data processing method according to the embodiment of the present disclosure is shown;

[0046] Figure 2 A second flowchart of a data processing method according to an embodiment of the present disclosure is shown;

[0047] Figure 3 A third flowchart of a data processing method according to an embodiment of the present disclosure is shown;

[0048] Figure 4One of the module structure schematic diagrams of a data processing device according to an embodiment of the present disclosure is shown;

[0049] Figure 5 A second schematic diagram of the module structure of the data processing device according to an embodiment of the present disclosure is shown;

[0050] Figure 6 The hardware structure block diagram of an electronic device capable of implementing the data processing method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with specific implementation methods and drawings. Here, the illustrative implementation methods and descriptions of the present disclosure are used to explain the present disclosure, but are not intended to limit the present disclosure.

[0052] As mentioned above, in the field of data processing, data files from different sources need to be adaptively configured and cleaned to meet the upload requirements of the data platform. The solution disclosed in this disclosure aims to solve the problem of cleaning and uploading data files from different sources in multiple scenarios. The following will combine an example to illustrate the application scenario of the solution disclosed in this disclosure:

[0053] In an example of the present disclosure, the scheme of the present disclosure can be applied to a data processing system / service, which can obtain data files uploaded by users, clean the data files, such as standardizing the data format, and upload the data files to a data platform through an upload interface for invocation by upstream services that rely on the data platform to provide data, wherein the data processing system / service can operate in coordination with a distributed coordination service, and in an embodiment of the present disclosure, the distributed coordination service is at least used to manage, update, and synchronize configuration files in the data processing system, and its functions include at least configuration management, configuration update, configuration synchronization, etc. The embodiment of the present disclosure does not specifically limit the distributed coordination service, and it is sufficient to be able to implement the functions of the distributed coordination service.

[0054] Figure 1 This is a method flow chart of an embodiment of the data processing method provided by the present disclosure, which can be applied to data processing systems / services including but not limited to the above embodiments of the present disclosure. The following description is made by taking the data processing service as an example.

[0055] like Figure 1 As shown, the data processing method of the embodiment of the present disclosure at least includes the following 110-140:

[0056] 110: receiving the data file to be uploaded, and extracting the header field of the data file to be uploaded.

[0057] In the embodiment of the present disclosure, the data file to be uploaded is, for example, sent by a user to a data processing service, and is processed by the data processing service and then uploaded to a target data platform.

[0058] In this embodiment, the "user" can be understood broadly, including any entity or system that sends the data file to be uploaded, including but not limited to individuals, organizations, cloud or local databases that upload data files, etc. Among them, the information, data and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions.

[0059] In the disclosed embodiment, the data processing service can extract the header field of the data file to be uploaded. As mentioned above, the method of the disclosed embodiment can be run in coordination with the distributed coordination service, and accordingly, the distributed coordination service can extract the header field of the data file.

[0060] In one example, extracting the header field of the data file to be uploaded may include:

[0061] The header fields in the data file to be uploaded are extracted through a target service application, wherein the target service application is, for example, a distributed coordination service application that provides consistency services for distributed applications.

[0062] In a specific example, the data processing service receives data file F1 uploaded by user A and data file F2 uploaded by user B. Then, the distributed coordination service extracts the header fields of the data files F1 and F2 to obtain:

[0063] The header fields of data file F1 are: “customName, custNum, reg_mob, aft_lmt, if_laohui, is_non_operate, a_risk, g_lent, bin_ascore, if_cs”;

[0064] The header fields of data file F2 are: “custNum, id, name, cell”.

[0065] In this embodiment, the header fields of the data files uploaded by different users (user A and user B) are different in terms of the fields included and the order of the fields. However, it is understandable that in other embodiments, the header fields of the data files to be uploaded by the same user in different scenarios may also be different, such as different fields or the order of the fields, which will not be repeated here.

[0066] In the embodiment of the present disclosure, a step of determining whether there is a configuration file matching the header field may also be included.

[0067] In an embodiment of the present disclosure, after extracting the header fields of the data file, the data processing service will check the existing configuration files to determine whether there is a configuration file that matches the header fields of the data file, wherein the configuration file may include a preset data field identifier set, a preset order of the data fields, etc.

[0068] In some embodiments of the present disclosure, for example, in the above 110, the data file to be uploaded may carry a user identifier, and accordingly, the configuration file may also include a user identifier. In this embodiment, the configuration file may include one or more user identifiers, and the user identifier is, for example, a unique user number, which is not limited by the present disclosure.

[0069] In a specific example, the coordination service in the data processing service stores the following configuration file 1:

[0070]

[0071] In this embodiment, the configuration file includes a user identifier "1000001", which is used to indicate that the configuration file was once applicable to the data file to be uploaded from the user "1000001"; a preset data field identifier set "demo", which includes 10 preset data field identifiers; an arrangement order "order" of the data fields corresponding to the data field identifiers in the data file, for example, the data field identifier "customName" corresponds to "0", which means that the configuration file will extract the "0"th data field from the data file to be uploaded and use it as the data field corresponding to the identifier "customName".

[0072] In some embodiments of the present disclosure, existing configuration files may be traversed to determine whether there is a configuration file that matches the header field.

[0073] Accordingly, the process of determining whether there is a configuration file matching the header field may include: matching the configuration file with the data field identifier in the header field to determine whether there is a configuration file adapted to the header field.

[0074] Continuing with the above specific example, the data processing service extracts the header field of the data file F1 sent by user A through the distributed coordination service, which is: "customName, custNum, reg_mob, aft_lmt, if_laohui, is_non_operate, a_risk, g_lent, bin_ascore, if_cs". Through the distributed coordination service search in the data processing service, it can be determined that the header field is adapted to the data field identifier "demo":"customName, custNum, cell, if_cs, aft_lmt, a_risk, g_lent, if_laohui, is_non_operate, bin_ascore" of the existing configuration file, that is, it is adapted to the configuration file.

[0075] As mentioned above, the data file to be uploaded and the configuration file may include a user identifier. In some embodiments of the present disclosure, it may be determined whether there is a configuration file adapted to a certain header field in combination with the user identifier. Accordingly, the process of determining whether there is a configuration file matching the header field may include the following steps S1-S3:

[0076] S1: Get the user ID in the data file to be uploaded.

[0077] In the embodiment of the present disclosure, the data processing service can parse and obtain the user identification in the data file to be uploaded. Examples of user identification can be found in the above embodiment and will not be described again here.

[0078] S2: Obtain one or more profiles associated with the user identity.

[0079] As mentioned above, the configuration files in the data processing service may include the user identifier, and thus, by traversing the existing configuration files, one or more configuration files having the user identifier in the data file to be uploaded may be determined as the configuration files associated with the user identifier.

[0080] In a specific example, the distributed coordination service in the data processing service stores the following configuration files 2 and 3:

[0081] Profile 2:

[0082]

[0083]

[0084] Profile 3:

[0085]

[0086] When searching for a configuration file, the data file 2 and the data file 3 both include the user ID "1000002", and the data file 2 and the data file 3 can be determined as configuration files associated with the user ID "1000002".

[0087] S3: Match one or more configuration files with the data field identifier in the header field to determine whether there is a configuration file that matches the header field.

[0088] In this embodiment, after obtaining one or more configuration files associated with the user identifier, the data field identifiers contained in the configuration files may be matched one by one with the data field identifiers in the header field to determine an adapted configuration file.

[0089] In a specific embodiment, referring to the description of the specific embodiment in S2 above, the header field of the data to be uploaded corresponding to the user identifier "1000002" is:

[0090] "customName,custNum,reg_mob,aft_lmt,if_laohui,is_non_operate,a_risk,g_lent,bin_ascore,if_cs".

[0091] The configuration files 2 and 3 associated with the user ID "1000002" are obtained. After matching, it is determined that the configuration file 2 includes the following "demo":

[0092] "customName,custNum,cell,if_cs,aft_lmt,a_risk,g_lent,if_laohui,is_non_operate,bin_ascore".

[0093] The data field identifier in "demo" of the configuration file 2 is adapted to the header field of the data file to be uploaded corresponding to the user identifier "1000002", so the configuration file 2 can be determined to be the configuration file adapted to the data to be uploaded.

[0094] 120: When it is determined that there is a configuration file matching the header field, extract the data field from the data file to be uploaded according to the configuration file and fill it into the interface upload document. The filled interface upload document is the target upload document.

[0095] In the disclosed embodiment, the interface upload document is a standardized document template, and the interface upload document includes multiple data items (dataItems), wherein each data item includes: data field identifier, data type of the data field, whether the data field must be transmitted, data length upper limit of the data field, data content represented by the data field, etc. It can be understood that the data fields corresponding to the multiple data items in the interface upload document can be filled from the user's data to be uploaded, and the filled interface upload document can be used as the target upload document.

[0096] By way of explanation and not limitation, in the embodiments of the present disclosure, for data files carrying different header fields, if one wants to upload the files through a unified data upload interface, a unified standardized format is required. Based on this, an interface upload document of the unified data upload interface can be set. The interface upload document can provide a standardized template to facilitate consistency processing of data files with different header fields. The data processing service can convert different header fields into a format that meets the requirements of the upload interface according to the configuration file.

[0097] In a specific example, as shown in Table 1 below, the interface upload document may include the following data items:

[0098] Table 1

[0099] Data field identification Data Types Maximum data length Data content represented cell String 100 Mobile phone number MD5 / SHA256 value; name String 100 User name MD5 / SHA256 value id String 100 Identity MD5 / SHA256 value custNum String 100 User unique number registerDate String 50

[0100] In this embodiment, the interface upload document includes five data field identifiers: "cell, name, id, custNum, registerDate", but it can be understood that in other embodiments, the interface upload document may also include a larger number of data field identifiers, and the present disclosure does not limit this.

[0101] In the embodiment of the present disclosure, as described above, the data processing service can convert different header fields into a format that meets the requirements of the upload interface according to the configuration file.

[0102] Specifically, when it is determined that there is a configuration file that matches the header field, the data processing service can extract data fields from the data file to be uploaded according to the configuration file and fill them into the interface upload document as the target upload document.

[0103] In a specific example, the interface document includes 10 data field identifiers:

[0104] “custNum,customName,cell,if_cs,aft_lmt,a_risk,g_lent,if_laohui,is_non_o perate,bin_ascore; the first one (i.e. the “0th” position) is “custNum”...the last one (i.e. the “9th” position) is “bin_ascore”.

[0105] The header fields of the data file F3 to be uploaded by user C are:

[0106] "customName,custNum,reg_mob,aft_lmt,if_laohui,is_non_operate,a_risk,g_lent,bin_ascore,if_cs";

[0107] The data processing service confirms that the configuration file adapted to the data file F1 is configuration file 3:

[0108]

[0109] In this embodiment, although the data file F3 to be uploaded already includes the 10 data fields required by the interface document, the arrangement order of these data fields in the data file F3 does not match the data fields in the interface document. For example, the "0" position of the header field of the data file F3 is "customName", while the first position in the data field identification sequence of the interface document is "custNum". Simply copying the data fields sequentially from the data file F3 will result in data filling errors in the interface document.

[0110] Therefore, at this time, the data can be cleaned according to "demo" and "order" in the configuration file (format standardization in this embodiment), and according to the mapping relationship between the data field identifier and the position of the data field in "order" in the configuration file 3, the data field is extracted from the corresponding position of the data file F1 and filled into the interface upload document:

[0111] Specifically, the data processing service can know from "demo" and "order" in configuration file 3 that: the first data field "custNum" in the interface document corresponds to the "0th" position in the data file F1, and thus the data field of the header field "customName" at the "0th" position in the data file F1 is extracted and filled into the data field customName required at the first position in the interface document... The data field of the header field "if_cs" at the "9th" position in the data file F1 is extracted and filled into the data field "if_cs" required at the fourth position in the interface document... The data field of the header field "if_cs" at the "8th" position in the data file F1 is extracted and filled into the data field "if_cs" required at the tenth position in the interface document, and the interface upload document after the 10 data fields are filled is used as the target upload document.

[0112] As mentioned above, the interface upload document contains multiple data statistics items, each data statistics item includes a data field identifier. Optionally, the data fields in the user's file to be uploaded have corresponding data field identifiers in the multiple data statistics items. At this time, for example, the interface upload document can be filled according to the configuration file according to the method described in 120.

[0113] In some other embodiments, for at least one data field in the data fields in the user's file to be uploaded, there is no corresponding data field identifier in the multiple data statistical items. In this case, the interface upload document cannot be filled according to the existing configuration file. In this regard, the embodiment of the present disclosure can fill the interface upload document through the following 130.

[0114] 130: When it is determined that there is no configuration file matching the header field, a configuration file is created for the header field, and data fields are extracted from the data file to be uploaded through the created configuration file and filled into the interface upload document as the target upload document.

[0115] In the embodiment of the present disclosure, for example, in the above 130, for a data file to be uploaded, when the data processing service does not find a configuration file that matches the header field of the data file, a new configuration file may be created.

[0116] Accordingly, in some embodiments of the present disclosure, Figure 2 As shown, the configuration file for the header field may include at least the following 131, 135, 136, 137:

[0117] 131: Obtain the transmission requirements of the data files to be uploaded uploaded by the operation node.

[0118] In some embodiments of the present disclosure, optionally, for a data file to be uploaded for which there is no matching configuration file, the data file will be partially transmitted to an upstream operation node of the data platform, and the operation node will check the data file and confirm the transmission requirement for the data file, specifically, the transmission requirement for the data field in the data file. In this embodiment, the file transmission requirement carries a data field identifier set to be uploaded for the data file to be uploaded.

[0119] By way of explanation and not limitation, if the data processing service does not find a configuration file that matches the header fields of the data file to be uploaded, this means that the data service platform has not processed this type of data file before. In this regard, it is difficult for the data processing service to accurately determine which data fields in the data file to be uploaded are required by the data platform.

[0120] As mentioned above, in the embodiment of the present disclosure, the oral upload document may include multiple data statistics items, wherein the multiple data statistics items may include, for example, data fields in the data field identification set in 131, and these data fields may be filled from the user's data to be uploaded.

[0121] 135: Get the header fields of the data file to be uploaded.

[0122] In the embodiment of the present disclosure, the content of the acquisition step in the above 135 can refer to the relevant description of extracting the header field of the data file to be uploaded in the above 110, and will not be repeated here.

[0123] 136: Searching for each data field identifier in the data field set in the header field of the data file to be uploaded, and determining the data position sequence of each data field in the data field set in the data file to be uploaded.

[0124] In the embodiment of the present disclosure, by way of explanation and not limitation, in combination with the relevant description of reference 120 above, the arrangement order of the data fields in the data file to be uploaded may not meet the requirements in the interface document. If directly uploaded, it will result in data filling errors in the interface document. Therefore, it is beneficial to determine the data position order of each data field in the data field set in the data file to be uploaded.

[0125] In a specific example, the data field set R includes 4 data field identifiers:

[0126] "custNum,customName,a_risk,cell";

[0127] The header fields of the data file F4 to be uploaded by user D are:

[0128] "customName,custNum,reg_mob,a_risk";

[0129] At this time, the data field identifier in the data field set can be searched in the header field of the data file F4 to be uploaded, and it can be determined that the first "custNum" in the data field set is located at the "1" position in the header field of the data file F4, the second "customName" in the data field set is located at the "0" position in the header field of the data file F4, the third "a_risk" in the data field set is located at the "3" position in the header field of the data file F4, and the fourth "cell" in the data field set is located at the "2" position in the header field of the data file F4.

[0130] Therefore, it can be determined that the data field set includes four data field identifiers: “custNum, customName, a_risk, cell” and the positions in the header field of the data file F4 to be uploaded of user D are “1, 0, 3, 2”.

[0131] In some embodiments, the data position sequence may be determined manually by the administrator of the data processing service. However, it is understandable that in other embodiments, the data position sequence of each data field in the data file to be uploaded may be automatically matched and determined based on the data characteristics of each data field, such as data length (length of numerical value), data type (number or Chinese character), etc., and the present disclosure does not limit this.

[0132] 137: According to the data field identification set and the determined data position sequence, a configuration file matching the header field of the data file to be uploaded is established.

[0133] In one example, the data field identification set and the data position sequence may be used as a configuration file for matching the header fields of the uploaded data file.

[0134] In a specific example, continuing with the above example, the data field identifiers included in the data field set R obtained in the above 136: "custNum, customName, a_risk, cell", and the determined positions of the data field set in the header field of the data file F4 to be uploaded by user D: "1, 0, 3, 2" can be used as the configuration file of the data file F4:

[0135] {

[0136] "demo":"custNum,customName,a_risk,cell",

[0137] "order":"1,0,3,2"

[0138] }.

[0139] However, it can be understood that the above embodiments of the present disclosure are only examples. In other embodiments, the obtained configuration file may also include other data such as the aforementioned user identification, and can be used to implement the above embodiments of the present disclosure, such as the methods involved in the above S1-S3, which will not be repeated here.

[0140] In the embodiment of the present disclosure, after a new configuration file is created for the data file to be uploaded, data fields can be extracted from the data file to be uploaded according to the newly created configuration file and filled into the interface upload document as the target upload document of the data file. In this embodiment, the relevant contents of obtaining the target upload document according to the configuration file can be referred to the description in the above 120, which will not be repeated here.

[0141] In the disclosed embodiment, the newly created configuration file may be stored in the data processing service, for example, in the distributed coordination service. In this way, the types of data files to be uploaded that the data processing service can adapt to may be continuously enriched.

[0142] In some embodiments of the present disclosure, optionally, considering that in actual operation, some data fields in the data field identification set in the acquired transmission requirements have no corresponding items in the data statistical items included in the interface upload document, for this reason, the data fields can be expanded by adding extended reserved fields to the interface upload document, so that the interface upload document is gradually improved to meet the needs of more users and scenarios.

[0143] Accordingly, in some embodiments of the present disclosure, before obtaining the header field of the data file to be uploaded, as Figure 3 As shown, the 130 may also include the following 132-134:

[0144] 132: Call the interface to upload the document.

[0145] In the embodiment of the present disclosure, the data processing service can retrieve the existing interface upload document and modify it. The specific description of the interface upload document can refer to the relevant description in the aforementioned 120 and will not be repeated here.

[0146] 133: traverse the data field identifier set to determine whether each data field identifier in the data field identifier set has been recorded in the interface upload document.

[0147] In an embodiment of the present disclosure, the data processing service may traverse the data field identification set in the acquired transmission requirement to determine whether all field identifications in the data field identification set are already included in the interface upload document.

[0148] In a specific example, the distributed coordination service of the data processing service obtains a data field identification set R1, which includes four data field identifications: "custNum, Name, id, cell", and the data statistics items shown in Table 2 are recorded in the interface upload document:

[0149] Table 2

[0150] Data item identification Data Types Parameter Value Corresponding data content cell String 100 Contact information MD5 / SHA256 value; name String 100 User name MD5 / SHA256 value id String 100 MD5 / SHA256 value of the certificate number custNum String 100 User unique number registerDate String 50 reserveField1 JSON Reserved Field 1 reserveField2 JSON Reserved Field 2

[0151] In this embodiment, the data field set R1 includes four data field identifiers: “custNum, Name, id, cell”, and corresponding data field identifiers can be found in the data statistics items.

[0152] In this embodiment, the interface upload document also includes two extended data statistics items "reserveField1" and "reserveField2", and these extended data statistics items can be customized, for example, referring to the description in the following 134, filling in additional data statistics items, etc. It can be understood that in other embodiments, the interface upload document can also include more extended data statistics items for use, and the present disclosure does not limit this.

[0153] 134: When it is determined that there are data field identifiers that are not recorded in the interface upload document, a corresponding data statistics item is added in the interface upload document for each unrecorded data field identifier.

[0154] In the embodiment of the present disclosure, for data field identifiers that are not recorded in the interface upload document, a new data statistics item may be added to the interface upload document.

[0155] In this embodiment, as described above, the interface upload document may include preset extended data statistics items. At this time, the unrecorded data field identifier may replace the content in the extended data statistics item to add a new data statistics item.

[0156] In a specific example, in combination with the example in the description of reference 133 and 134, the distributed coordination service of the data processing service obtains a data field identification set R2, wherein the data field set R1 includes four data field identifications: "custNum, Name, id, cell, source", wherein the four items "custNum", "Name", "id", and "cell" can all find corresponding data field identifications in the interface upload document shown in the aforementioned Table 2, but "source" does not have a corresponding data field identification. For this, the extended data statistics item "reserveField1" can be modified in the interface upload document to add a new data statistics item. The modified interface upload document is shown in Table 3 below:

[0157] Table 3

[0158] Data item identification Data Types Parameter Value Corresponding data content cell String 100 Contact information MD5 / SHA256 value; name String 100 User name MD5 / SHA256 value id String 100 MD5 / SHA256 value of the certificate number custNum String 100 User unique number registerDate String 50 source String Data Source reserveField2 JSON Reserved Field 2

[0159] Among them, the reserved extended data statistical item "reserveField1" is changed to "source".

[0160] However, it is understandable that, in another embodiment, when the extended data statistics item does not exist, a new line may be directly created in the interface upload document to fill in the new data statistics item, and the present disclosure does not impose any restrictions on this.

[0161] In some embodiments of the present disclosure, when or after adding a corresponding data statistic item in the interface upload document for each unrecorded data field identifier, the method of the present disclosure may further include the following steps:

[0162] The original interface upload document is updated through the target service application, and the original interface upload document is replaced with the interface upload document to which the corresponding data statistics item is added.

[0163] In this embodiment, the target service application includes a distributed coordination service application that provides consistency services for distributed applications. The original interface upload document is, for example, the existing interface upload document in the above 132, and the interface upload document with the corresponding data statistics items added is, for example, the interface upload document with the new data statistics items added obtained through the above 134.

[0164] In this embodiment, the interface upload document can be updated based on the distributed coordination service: a new interface upload document with expanded data fields is used to replace the original interface upload document. This method enriches the types of data fields applicable to the interface upload document, thereby being able to meet the needs of more different types of data files.

[0165] 140: Upload the target upload document to the target data platform through the data upload interface.

[0166] In the disclosed embodiment, the interface upload document after filling in the data fields in the user's data file to be uploaded can be used as the target upload document. Any user's data file to be uploaded is converted into a target upload document of the same format, and the target upload document is uploaded to the target data platform through the data upload interface. The data platform can directly read the target upload document to obtain the data fields. The update of the configuration file is decoupled from the development of the upload interface. When the user's data file changes, only a new configuration file needs to be created. There is no need to repeatedly develop and debug the upload interface, which improves the efficiency of the data platform in acquiring data.

[0167] In the disclosed embodiment, a data file to be uploaded can be received, and the header fields of the data file to be uploaded can be extracted; it is determined whether there is a configuration file matching the header fields; if it is determined that there is a configuration file matching the header fields, the data fields are extracted from the data file to be uploaded according to the configuration file and filled into the interface upload document, and the filled interface upload document is the target upload document; if it is determined that there is no configuration file matching the header fields, a configuration file is established for the header fields, and the data fields are extracted from the data file to be uploaded and filled into the interface upload document through the established configuration file, and the filled interface upload document is the target upload document; the target upload document is uploaded to the target data platform through the data upload interface. The method of the disclosed embodiment solves the technical problem that the upload interface needs to be developed multiple times due to the incompatibility of the existing header fields, and the data maintenance cost is too high, and achieves the compatible upload of data fields in different data files of different users, as well as the technical effect of reducing the workload of development and reducing the data maintenance cost.

[0168] Accordingly, in some embodiments of the present disclosure, Figure 4 As shown, a data processing device 400 is also provided, which may include a receiving module 410, a filling module 420, a creating module 430 and an uploading module 440, wherein:

[0169] The receiving module 410 is configured to receive a data file to be uploaded and extract a header field of the data file to be uploaded.

[0170] The filling module 420 is configured to extract data fields from the data file to be uploaded according to the configuration file and fill them into the interface upload document when it is determined that there is a configuration file matching the header field, and the filled interface upload document is the target upload document.

[0171] The establishment module 430 is configured to establish a configuration file for the header field when it is determined that there is no configuration file matching the header field, and extract data fields from the data file to be uploaded through the established configuration file and fill them into the interface upload document. The filled interface upload document is the target upload document.

[0172] The upload module 440 is configured to upload the target upload document to the target data platform through the data upload interface.

[0173] In some embodiments of the present disclosure, Figure 4 and Figure 5 , the establishment module 430 of the data processing device 400 may include a first acquisition unit 431, a second acquisition unit 432, a search unit 433 and a generation unit 434, wherein:

[0174] The first acquisition unit 431 is configured to acquire a transmission requirement for the data file to be uploaded uploaded by the operation node, wherein the file transmission requirement carries a data field identifier set required to be uploaded for the data file to be uploaded;

[0175] The second acquisition unit 432 is configured to acquire the header field of the data file to be uploaded;

[0176] The search unit 433 is configured to search for each data field identifier in the data field set in the header field of the data file to be uploaded, so as to determine the data position sequence of each data field in the data field set in the data file to be uploaded.

[0177] The generating unit 434 is configured to establish a configuration file that matches the header field of the data file to be uploaded according to the data field identification set and the determined data position sequence.

[0178] In an embodiment of the present disclosure, an electronic device is also provided, including: a processor and a memory storing a computer program, wherein the processor is configured to execute any method of searching for web page resources in the embodiments of the present disclosure when running the computer program.

[0179] Figure 6 A schematic diagram of an electronic device 600 that can implement the method of an embodiment of the present disclosure or implement an embodiment of the present disclosure is shown, and in some embodiments, more or fewer electronic devices than shown may be included. In some embodiments, it can be implemented using a single or multiple electronic devices. In some embodiments, it can be implemented using cloud or distributed electronic devices. The method embodiments provided in the above embodiments of the present disclosure can be executed in a mobile terminal, a computer terminal, or a similar computing device.

[0180] like Figure 6 As shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 602 or the programs and / or data loaded from the storage part 608 to the random access memory (RAM) 603. CPU 601 can be a multi-core processor, or it can include multiple processors. In some embodiments, CPU 601 can include a general main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. CPU 601, ROM 602 and RAM 603 are connected to each other via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0181] The processor and the memory are used together to execute the program stored in the memory, and when the program is executed by the computer, the steps or functions of the method described in the above embodiments of the present disclosure can be implemented.

[0182] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, a touch screen, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed. Figure 6 Only some components are schematically shown in the figure, which does not mean that the computer system 600 only includes Figure 6 Components shown.

[0183] In some embodiments, the electronic device 600 refers to a mobile terminal or a computer, including a mobile phone, a car terminal, a smart TV, etc. Taking a mobile phone as an example, the electronic device 600 also includes a display screen with a touch function, an external speaker, a gyroscope, a camera, a 4G / 5G antenna and other device modules.

[0184] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0185] Although not shown, in an embodiment of the present disclosure, a storage medium is provided, which stores a computer program. When the computer program is executed by a processor, all the steps of the data processing method in the above-mentioned embodiment of the present disclosure are implemented. For example, when the processor executes the computer program, the following steps are implemented.

[0186] The storage media in the embodiments of the present disclosure include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0187] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0188] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] Although the present specification embodiment provides the method operation steps as described in the embodiment or flow chart, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way in the order of execution of many steps, and does not represent a unique execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel (such as a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment) according to the method shown in the embodiment or the accompanying drawings. The term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements.

[0190] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0192] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0194] The present specification embodiments may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification embodiments may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0195] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiments of this specification. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0196] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. A data processing method, characterized in that: include: Receiving a data file to be uploaded, and extracting a header field of the data file to be uploaded; If it is determined that there is a configuration file matching the header field, extract the data field from the data file to be uploaded according to the configuration file and fill it into the interface upload document, and the filled interface upload document is the target upload document; In the case where it is determined that there is no configuration file matching the header field, a configuration file is created for the header field, and data fields are extracted from the data file to be uploaded through the created configuration file and filled into the interface upload document, and the filled interface upload document is used as the target upload document; The target upload document is uploaded to the target data platform through the data upload interface.

2. The data processing method according to claim 1, characterized in that: The step of establishing a configuration file for the header field includes: Acquire a transmission requirement uploaded by an operation node for the data file to be uploaded, wherein the file transmission requirement carries a data field identifier set that needs to be uploaded for the data file to be uploaded; Obtaining the header fields of the data file to be uploaded; Searching the identifiers of each data field in the data field set in the header field of the data file to be uploaded to determine the data position sequence of each data field in the data field set in the data file to be uploaded; A configuration file matching the header field of the data file to be uploaded is established according to the data field identification set and the determined data position sequence.

3. The data processing method according to claim 2, characterized in that: After obtaining the transmission requirement for the data file to be uploaded uploaded by the operation node, the method further includes: Retrieve an interface upload document, wherein the interface upload document includes a plurality of data statistics items, wherein each data statistics item includes: a data field identifier, a data type of the data field, an upper limit of a data length of the data field, and data content represented by the data field; Traversing the data field identification set to determine whether each data field identification in the data field identification set has been recorded in the interface upload document; In the case where it is determined that there are data field identifiers that are not recorded in the interface upload document, a corresponding data statistic item is added to the interface upload document for each unrecorded data field identifier.

4. The data processing method according to claim 3, characterized in that: The corresponding data statistics items are added to the interface upload document for each unrecorded field identifier, including: In the reserved field position in the interface upload document, for each field identifier that is not recorded, a corresponding data statistics item is added in the interface upload document.

5. The data processing method according to any one of claims 1 to 4, characterized in that: The data file to be uploaded carries a user identifier; Then, determining whether there is a configuration file matching the header field includes: Obtaining a user ID from the data file to be uploaded; Obtaining one or more configuration files associated with the user identifier; The one or more configuration files are matched with the data field identifier in the header field to determine whether there is a configuration file adapted to the header field.

6. The data processing method according to claim 3 or 4, characterized in that: The method further comprises: The original interface upload document is updated through the target service application, and the original interface upload document is replaced with the interface upload document to which the corresponding data statistics item is added, wherein the target service application is a service application that provides consistency service for distributed applications.

7. A data processing device, characterized in that: include: A receiving module, configured to receive a data file to be uploaded and extract a header field of the data file to be uploaded; A filling module is configured to extract data fields from the data file to be uploaded according to the configuration file and fill them into the interface upload document when it is determined that there is a configuration file matching the header field, and the interface upload document that has been filled is the target upload document; an establishing module, configured to establish a configuration file for the header field if it is determined that there is no configuration file matching the header field, and extract data fields from the data file to be uploaded through the established configuration file and fill them into the interface upload document, and the filled interface upload document is the target upload document; The upload module is configured to upload the target upload document to the target data platform through the data upload interface.

8. The data processing device according to claim 7, characterized in that: The establishment module includes: A first acquisition unit is configured to acquire a transmission requirement for the data file to be uploaded uploaded by the operation node, wherein the file transmission requirement carries a data field identifier set that needs to be uploaded for the data file to be uploaded; A second acquisition unit is configured to acquire a header field of the data file to be uploaded; A search unit configured to search for each data field identifier in the data field set in the header field of the data file to be uploaded, so as to determine the data position sequence of each data field in the data field set in the data file to be uploaded; The generating unit is configured to establish a configuration file matching the header field of the data file to be uploaded according to the data field identification set and the determined data position sequence.

9. An electronic device comprising a processor and a memory for storing instructions executable by the processor, characterized in that: When the processor executes the instructions, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images