Data processing method, device and server

Abstract

This specification provides a data processing method, device and server. Based on this method, when the data lake server detects that it has received target data updated and transmitted by the upstream application, it can first pre-process the target data before loading the target data into the target data lake; when it is determined that the target data pre-processing check has passed, the target data is then loaded into the target data lake in a source-closed manner to ensure that the data loaded into the target data lake is reliable and consistent with the upstream system; then the downstream application associated with the target data is obtained and determined based on the blood relationship of the target data lake; the downstream application is called to batch load and / or process the target data. In this way, data processing related to the data lake can be completed automatically and efficiently, ensuring that the data loaded into the lake is accurate and reliable, and the corresponding downstream applications can be called in a timely manner to complete the loading and / or processing of the new data entering the lake in batches.

Description

Technical Field

[0001] This specification belongs to the field of big data technology, and in particular, relates to data processing methods, devices and servers. Background Art

[0002] In the field of big data technology, data lakes are often used to store and share massive amounts of business data accessed by a system or platform.

[0003] However, based on the existing data processing methods, when performing relevant data processing on the data lake, there are often technical problems such as cumbersome and complex data processing process and low processing efficiency.

[0004] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0005] This specification provides a data processing method, device, and server to automatically and efficiently complete data processing related to a data lake.

[0006] The embodiment of this specification provides a data processing method, which is applied to a data lake server, including:

[0007] Check whether the target data transmitted by the upstream application is received;

[0008] When it is determined that the target data transmitted by the upstream application is received, the target data is pre-processed and checked; and the service returned by the service provider is received.

[0009] When it is determined that the target data preprocessing check has passed, the target data is loaded into the target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications;

[0010] Obtain and determine the downstream applications associated with the target data based on the lineage relationship of the target data lake;

[0011] The downstream application is called to batch load and / or process the target data.

[0012] In some embodiments, the target data also carries a download time.

[0013] In some embodiments, after determining that the target data downloaded by the upstream application is received, the method further includes:

[0014] Obtain the download time of the target data downloaded by the upstream application with the same application identifier as received last time as a reference time;

[0015] By comparing the reference time with the download time of the target data, it is determined whether the target data is valid.

[0016] In some embodiments, the target data is pre-processed and checked, including:

[0017] According to the preset data inspection rules, the target characteristics of the target data are inspected to determine whether the target characteristics meet the preset requirements; wherein the target characteristics include at least one of the following: the length of the data, the code system of the data, the name of the data, the transmission information of the data, the structure of the data, the storage path of the data, and the storage format of the data;

[0018] When it is determined that the target features meet the preset requirements, it is determined that the target data preprocessing check has passed.

[0019] In some embodiments, the target data is loaded into the target data lake in a source-to-lake manner, including:

[0020] Get data quality check rules that match the upstream application;

[0021] While the target data is loaded into the target data lake, key data is extracted from the target data entering the lake according to the data quality check rules, and a data quality check is performed on the key data so that the target data entering the lake matches the upstream application.

[0022] In some embodiments, after calling the downstream application to batch load and / or process the target data, the method further includes:

[0023] Determining whether a processing result of calling the downstream application to batch load and / or process the target data is a failure;

[0024] When the processing result is determined to be a failure, an error log is generated;

[0025] Sending the error log to the downstream application.

[0026] In some embodiments, the target data includes an interface file.

[0027] In some embodiments, the upstream application includes a bank card application; the downstream application includes an account risk control application.

[0028] The embodiment of this specification also provides a data processing device, which is applied to a data lake server, including:

[0029] A detection module is used to detect whether the target data transmitted by the upstream application is received;

[0030] A checking module, configured to perform a pre-processing check on the target data when it is determined that the target data is received and downloaded from the upstream application;

[0031] A loading module, used to load the target data into a target data lake in a source-to-lake manner when it is determined that the target data preprocessing check has passed; wherein the target data lake is used to store data shared by upstream applications and downstream applications;

[0032] The determination module is used to obtain and determine the downstream applications associated with the target data based on the lineage relationship of the target data lake;

[0033] A calling module is used to call the downstream application to batch load and / or process the target data.

[0034] The embodiment of the present specification also provides a server, including a processor and a memory for storing processor executable instructions, wherein the processor implements the following contents when executing the instructions: detecting whether target data transmitted by an upstream application is received; when it is determined that the target data transmitted by an upstream application is received, performing a preprocessing check on the target data; when it is determined that the target data preprocessing check passes, loading the target data into a target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications; obtaining and determining downstream applications associated with the target data based on the blood relationship of the target data lake; and calling the downstream applications to batch load and / or process the target data.

[0035] The embodiments of the present specification also provide a computer-readable storage medium having computer instructions stored thereon, which when executed implement the following: detecting whether target data transmitted by an upstream application is received; if it is determined that the target data transmitted by an upstream application is received, performing a preprocessing check on the target data; if it is determined that the target data preprocessing check passes, loading the target data into a target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications; obtaining and determining downstream applications associated with the target data based on the blood relationship of the target data lake; and calling the downstream applications to batch load and / or process the target data.

[0036] This specification provides a data processing method, device and server. Based on this method, when the data lake server detects that it has received target data updated and transmitted by an upstream application, it can first pre-process the target data before loading the target data into the target data lake; when it is determined that the target data pre-processing check has passed, the target data is then loaded into the target data lake in a source-attached manner to ensure that the data loaded into the target data lake is accurate and reliable, and consistent with the upstream system; then the downstream application associated with the target data is obtained and determined based on the blood relationship of the target data lake; and then the downstream application is called in a targeted manner to batch load and / or process the target data. This can effectively simplify data processing related to the data lake, realize automatic and efficient completion of data processing related to the data lake, ensure that the data loaded into the lake is accurate and reliable, and can timely and targetedly call the corresponding downstream application to batch complete the loading and / or processing of the new data entering the lake, thereby improving the overall processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flowchart of a data processing method provided by an embodiment of this specification;

[0039] Figure 2 It is a schematic diagram of the structure of a server provided by an embodiment of this specification;

[0040] Figure 3 is a schematic diagram of the structure of a data processing device provided by an embodiment of this specification;

[0041] Figure 4 It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example;

[0042] Figure 5 It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example;

[0043] Figure 6 It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example;

[0044] Figure 7It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example;

[0045] Figure 8 It is a schematic diagram of an embodiment of applying the data processing method provided by the embodiments of this specification in a scenario example. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0047] See also Figure 1 As shown, the embodiment of this specification provides a data processing method, wherein the method is specifically applied to the data lake server side. When implemented specifically, the method may include the following contents:

[0048] S101: Detect whether target data transmitted by an upstream application is received;

[0049] S102: When it is determined that target data downloaded from an upstream application is received, pre-processing and checking the target data; and receiving a service returned by the service provider.

[0050] S103: If it is determined that the target data preprocessing check has passed, the target data is loaded into the target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications;

[0051] S104: Obtain and determine downstream applications associated with the target data based on the blood relationship of the target data lake;

[0052] S105: Call the downstream application to batch load and / or process the target data.

[0053] Through the above embodiments, the data lake server can timely discover the target data updated and downloaded by the upstream application; and before loading the target data into the lake, the target data is pre-processed and checked; if the pre-processing check passes, the target data is loaded into the target data lake in a source-closed manner to ensure that the target data loaded into the lake is accurate and reliable, and is consistent and matches the data of the upstream application; further, the data lake server can determine the downstream application associated with the target data based on the blood relationship of the target data lake, and then specifically call the corresponding job threads of the downstream application to batch load and / or process the target data, thereby effectively simplifying the data processing related to the data lake, improving the overall data processing efficiency, and realizing automatic and efficient completion of data processing related to the data lake.

[0054] In some embodiments, the method can be specifically applied to a data lake server, wherein the data lake server can be specifically deployed on one side of the target data lake to manage and schedule the server of the target data lake.

[0055] In this embodiment, the data lake server may specifically include a background server applied to one side of the data processing system, which can realize functions such as data transmission and data processing. Specifically, the data lake server may be, for example, an electronic device with data calculation, storage and network interaction functions. Alternatively, the data lake server may also be a software program running in the electronic device to provide support for data processing, storage and network interaction. In this embodiment, the number of servers included in the data lake server is not specifically limited. The data lake server may specifically be one server, several servers, or a server cluster formed by several servers.

[0056] In this implementation, the target data lake (or target data lake table) is specifically understood as a centralized data repository for business data storage and data sharing.

[0057] Among them, the upstream application and the downstream application can be connected to the target data lake respectively. Specifically, the upstream application can transfer the updated business data to the target data lake for storage. The downstream application can obtain the updated business data through the target data lake for corresponding use and processing.

[0058] In some embodiments, the target data may be specifically understood as newly updated business data of the upstream application to be downloaded and stored in the target data lake, for example, newly updated basic data information.

[0059] In some embodiments, the data lake server may be mounted on a server for downloading and storing the upstream application (referred to as the upstream server). Accordingly, the data lake server may interact with the server for downloading and storing the upstream application mounted thereon to detect whether the target data downloaded by the upstream application is received.

[0060] In some embodiments, the target data may include structured data and / or unstructured data. For different application scenarios, the target data may include business data of different types and contents.

[0061] In some embodiments, taking the transaction data processing scenario of a bank as an example, the target data may specifically include an interface file of an upstream application. The specific content of the target data may include one or more of the following data: user information of a user, transaction information of an account, institution information of a bank, etc.

[0062] Specifically, the target data may be a data composed of three data formats: numeric, character, and date. Specifically, taking the user's fund transaction record in a bank as the target data, it can be represented as data of the following structure: [name + card number + amount + transaction type + transaction amount + outlet].

[0063] In this embodiment, the upstream application may specifically include a bank card application, etc. Correspondingly, the downstream application may specifically include an account risk control application, etc. The account risk control application specifically refers to an account transaction risk control application.

[0064] Of course, it should be noted that the upstream applications and downstream applications listed above are only schematic illustrations. In specific implementations, the upstream applications may also include one or more functional applications such as deposit applications, loan applications, and financial management applications in addition to bank card applications. Similarly, the downstream applications may also include one or more functional applications such as marketing applications, customer relationship management applications, and information push applications in addition to account risk control applications.

[0065] Through the above embodiments, the data processing method provided in the embodiments of this specification can be better applied to the transaction processing scenarios of banks to effectively load and process various target data between various upstream applications and downstream applications.

[0066] In addition to being applied to transaction data processing scenarios in banks, the data processing method can also be further applied to other types of application scenarios (for example, customer service response scenarios, etc.) to process relevant data from other scenarios.

[0067] It should be added that in the field of big data technology, for application scenarios such as bank transaction data processing scenarios, the target data faced by the data lake server may often be a large amount of business data from multiple upstream servers at the same time, and there are also correlations between them. For the above massive target data, based on the existing data processing methods, it is often difficult to efficiently and automatically realize the data loading and processing into the lake, and errors are prone to occur during the processing based on the existing data processing methods.

[0068] In some embodiments, the target data may further include a download time.

[0069] Specifically, when the upstream server downloads the target data, it generates a timestamp according to the download time, and adds the timestamp to the target data, so that the target data can carry the corresponding download time.

[0070] Through the above embodiment, the data lake server can determine the download time of the target data according to the timestamp carried in the target data, and then judge whether the target data is the latest business data according to the download time, so as to determine whether to load the target data into the target data lake.

[0071] In some embodiments, after determining that the target data downloaded by the upstream application is received, the method may also include the following contents when implemented: obtaining the download time of the target data downloaded by the upstream application with the same application identifier as the last received data as a reference time; determining whether the target data is valid by comparing the reference time with the download time of the target data.

[0072] In specific implementation, when the data lake server determines through comparison that the reference time is later than the download time of the target data, it can determine that the target data is invalid data that has expired, and then directly refuse to load the target data into the target data lake for storage. On the contrary, when the data lake server determines through comparison that the reference time is earlier than the download time of the target data, it can determine that the target data is valid data that has not expired, and then trigger subsequent data processing, so that the target data can be automatically and efficiently loaded and stored in the target data lake.

[0073] Through the above embodiment, it is possible to timely identify and determine expired invalid data according to the download time carried by the target data, and stop wasting data processing resources for subsequent data processing on the expired invalid data. This can effectively reduce the amount of data processing on the one hand, and avoid data errors introduced by loading and storing expired invalid data into the target data lake on the other hand.

[0074] In some embodiments, the target data is pre-processed and checked. When implemented specifically, the following contents may be included: according to preset data checking rules, the target characteristics of the target data are checked to determine whether the target characteristics meet the preset requirements; wherein the target characteristics include at least one of the following: the length of the data, the coding of the data, the name of the data, the transmission information of the data, the structure of the data, the storage path of the data, and the storage format of the data; when it is determined that the target characteristics meet the preset requirements, it is determined that the target data pre-processing check has passed.

[0075] In this embodiment, when it is implemented, it is determined that the target data preprocessing check has passed and there is no data error only when the target features meet the preset requirements according to the preset data inspection rules. Accordingly, the subsequent loading process of the target data into the lake can be triggered.

[0076] On the contrary, if it is determined that at least one of the target features does not meet the preset requirements, it can be determined that the target data preprocessing check fails and there are data errors. Accordingly, the target data can be returned to the upstream server, and a prompt message of data error is generated and sent to the upstream server to prompt the upstream server to check and correct the target data and then re-download the target data without data errors.

[0077] Through the above embodiments, before loading and storing the target data into the target data lake, the data lake server can pre-process the target data to promptly discover the target data with data errors, and then promptly return it to the upstream server for processing to ensure that the target data subsequently loaded into the lake is accurate and reliable.

[0078] In some embodiments, before the target data is loaded and stored in the target data lake, the target data may be pre-processed and checked according to preset data checking rules, so that the data subsequently loaded and stored in the target data lake is accurate and meets the requirements. The preset data checking rules may specifically include multiple target features to be checked, as well as preset requirements for each target feature.

[0079] In some embodiments, the above-mentioned loading of the target data into the target data lake in a source-to-lake manner may include the following contents during specific implementation:

[0080] S1: Obtain data quality check rules that match the upstream application;

[0081] S2: While loading the target data into the target data lake; according to the data quality check rules, extract key data from the target data entering the lake, and perform a data quality check on the key data, so that the target data entering the lake matches the upstream application.

[0082] Through the above embodiments, the data lake server can perform data quality checks on the data loaded into the target data lake while loading and storing the target data into the target data lake according to data quality check rules that match the upstream application, so as to ensure that the data loaded into the lake is accurate and consistent with and matches the upstream application, and further ensure that the target data loaded into the lake is accurate and reliable.

[0083] In some embodiments, before specific implementation, a plurality of corresponding preset data quality check rules may be configured for each upstream application connected to the target data lake, and a correspondence between the preset data quality check rules and the application identifiers of the corresponding upstream applications may be established.

[0084] During specific implementation, multiple preset data quality check rules may be first obtained and retrieved according to the application identifier of the upstream application to find a matching data quality check rule.

[0085] During specific implementation, the key data extracted for inspection may specifically include: primary key data, key fields, etc.

[0086] By timely checking the target data entering the lake according to the data quality check rules, it is possible to carry out targeted data quality checks on the target data entering the lake, such as error row fault tolerance compatibility, file primary key, etc., to further ensure that the target data loaded into the lake is accurate and reliable.

[0087] In some embodiments, the above-mentioned blood relationship can be specifically understood as a data dependency relationship between an upstream application and a downstream application.

[0088] Specifically, a blood relationship table may be stored in the target data lake, and blood relationships between multiple upstream applications and downstream applications connected to the target data lake may be recorded based on the blood relationship table.

[0089] In specific implementation, the blood relationship table can be searched according to the application identifier of the upstream application to obtain the blood relationship including the application identifier of the upstream application. Further, the downstream application associated with the target data can be determined according to the above blood relationship.

[0090] In some embodiments, during specific implementation, the full-volume job thread of the downstream application may be called to load the target data in batches and / or process the target data in a targeted manner, so as to ensure that the target data can be timely and efficiently transmitted to the corresponding downstream application.

[0091] In some embodiments, after calling the downstream application to batch load and / or process the target data, the method may further include the following when implemented:

[0092] S1: Determine whether a processing result of calling the downstream application to batch load and / or process the target data is a failure;

[0093] S2: when it is determined that the processing result is a failure, an error log is generated;

[0094] S3: Send the error log to the downstream application.

[0095] Through the above embodiments, the data lake server can determine that the current call has failed when it is found that the processing result of calling the downstream application to batch load and / or process the target data is a failure; and then can promptly generate a corresponding error log and send the error log to the downstream application, so that the downstream application can check the specific cause of the call failure according to the error log, so that the downstream application can better cooperate with the data lake server to successfully complete the batch loading and / or processing of the target data.

[0096] In some embodiments, when it is determined that the target data preprocessing check fails, a corresponding error log may be generated and sent to the upstream application, so that the upstream application can find out the reason why the preprocessing check fails according to the error log, and then make targeted corrections to the target data, and then re-transmit the target data that accurately meets the preset requirements to the data lake server.

[0097] As can be seen from the above, based on the data processing method provided in the embodiment of this specification, when the data lake server detects that it has received target data updated and transmitted by the upstream application, it can first perform a pre-processing check on the target data before loading the target data into the target data lake; when it is determined that the target data pre-processing check has passed, the target data is then loaded into the target data lake in a source-attached manner to ensure that the data loaded into the target data lake is reliable and consistent with the upstream system; then the downstream application associated with the target data is obtained and determined based on the blood relationship of the target data lake; the downstream application is called to batch load and / or process the target data. In this way, data processing related to the data lake can be completed automatically and efficiently, ensuring that the data loaded into the lake is accurate and reliable, and the corresponding downstream applications can be called in a timely manner to complete the loading and / or processing of the new data entering the lake in batches.

[0098] The embodiment of the present specification also provides a server, including a processor and a memory for storing processor executable instructions. When the processor is implemented, it can perform the following steps according to the instructions: detect whether the target data transmitted by the upstream application is received; if it is determined that the target data transmitted by the upstream application is received, perform a pre-processing check on the target data; if it is determined that the target data pre-processing check passes, load the target data into a target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications; obtain and determine the downstream application associated with the target data based on the blood relationship of the target data lake; and call the downstream application to batch load and / or process the target data.

[0099] In order to complete the above instructions more accurately, refer to Figure 2 As shown, the embodiment of this specification also provides another specific server, wherein the server includes a network communication port 201, a processor 202 and a memory 203, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0100] The network communication port 201 can be used to receive target data downloaded from an upstream application.

[0101] The processor 202 can be specifically used to detect whether the target data transmitted by the upstream application is received; if it is determined that the target data transmitted by the upstream application is received, pre-process the target data; if it is determined that the target data pre-processing check is passed, the target data is loaded into the target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications; obtain and determine the downstream application associated with the target data based on the blood relationship of the target data lake; and call the downstream application to batch load and / or process the target data.

[0102] The memory 203 may be specifically used to store corresponding instruction programs.

[0103] In this embodiment, the network communication port 201 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0104] In this embodiment, the processor 202 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.

[0105] In this embodiment, the memory 203 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function that has no physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0106] The embodiments of this specification also provide a computer-readable storage medium based on the above data processing method, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the following are achieved: detecting whether target data transmitted by an upstream application is received; if it is determined that target data transmitted by an upstream application is received, performing a preprocessing check on the target data; if it is determined that the target data preprocessing check passes, loading the target data into a target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications; obtaining and determining downstream applications associated with the target data based on the blood relationship of the target data lake; and calling the downstream applications to batch load and / or process the target data.

[0107] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0108] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other implementations and will not be described in detail here.

[0109] See also Figure 3 As shown, at the software level, the embodiment of this specification also provides a data processing device, which may specifically include the following structural modules:

[0110] The detection module 301 may be specifically used to detect whether the target data transmitted by the upstream application is received;

[0111] The inspection module 302 may be specifically configured to perform a pre-processing inspection on the target data when determining that the target data is received from the upstream application;

[0112] The loading module 303 may be specifically used to load the target data into the target data lake in a source-to-lake manner when it is determined that the target data preprocessing check has passed; wherein the target data lake is used to store data shared by upstream applications and downstream applications;

[0113] The determination module 304 may be specifically used to obtain and determine downstream applications associated with the target data based on the lineage relationship of the target data lake;

[0114] The calling module 305 may be specifically used to call the downstream application to batch load and / or process the target data.

[0115] In some embodiments, the target data may also specifically carry a download time.

[0116] In some embodiments, the device may further include a timeliness judgment module, which may be specifically used to obtain the download time of the target data downloaded by the upstream application with the same application identifier as the last received, as a reference time; by comparing the reference time with the download time of the target data, it is determined whether the target data is valid.

[0117] In some embodiments, when the detection module 302 is implemented, it can be executed according to the following procedures: according to the preset data inspection rules, the target characteristics of the target data are checked to determine whether the target characteristics meet the preset requirements; wherein the target characteristics include at least one of the following: the length of the data, the coding of the data, the name of the data, the transmission information of the data, the structure of the data, the storage path of the data, and the storage format of the data; when it is determined that the target characteristics meet the preset requirements, it is determined that the target data preprocessing check has passed.

[0118] In some embodiments, when the above-mentioned loading module 303 is implemented, it can be executed according to the following procedures: obtaining data quality check rules that match the upstream application; while loading the target data into the target data lake; according to the data quality check rules, extracting key data from the target data entering the lake, and performing data quality check on the key data, so that the target data entering the lake matches the upstream application.

[0119] In some embodiments, the device may further include a log module for determining whether the processing result of calling the downstream application to batch load and / or process the target data is a failure; if the processing result is determined to be a failure, generating an error log; and sending the error log to the downstream application.

[0120] In some embodiments, the target data may specifically include an interface file.

[0121] In some embodiments, the upstream application may specifically include a bank card application; the downstream application may specifically include an account risk control application.

[0122] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] It can be seen from the above that the data processing device provided in the embodiments of this specification can automatically and efficiently complete data processing related to the data lake, ensure that the data loaded into the lake is accurate and reliable, and can promptly call the corresponding downstream applications to complete the loading and / or processing of new data entering the lake in batches.

[0124] In a specific scenario example, the data processing method provided in the embodiments of this specification can be applied to automatically batch load and test interface files. The specific implementation process may include the following contents.

[0125] In this scenario example, the cloud service platform based on big data can usually support the storage and open sharing of PB-level massive data, and can provide efficient, cost-controlled, and dynamically allocated enterprise-level big data services, providing stronger support for the integration, data governance, and application services of the group data. Among them, the data lake is a sharing platform for basic data information in the cloud service platform. More and more data is entering the lake. In order to ensure that the data in the test environment is the latest as soon as possible, to speed up the correctness and freshness of the data entering the lake for testing, and to ensure that downstream applications use the latest basic data information of the data lake as soon as possible, the data processing method provided in the embodiment of this application is used to perform relevant data processing of the data lake.

[0126] In this scenario example, the interface file (for example, the target data) can be detected in an automated way first. The interface file is a network transmission file that stores data. The file name ends with a date and is used to indicate the data to be loaded every day. The data in the file is executed by the loading program, and the data in a fixed format is loaded into the target data lake table (for example, the target data lake) specified by the data lake. The data lake table partition and the interface file time are consistent. The data loading method of the data lake table is to load the basic data information of the interface file at the source, which can ensure the integrity and freshness of the data in the data lake. When the data in the data lake table is loaded successfully, the blood relationship chain (or blood relationship) can be called through the table in the lake, and the latest data in the lake can be dispatched through the whole process, and the updated data lake of the data lake can be timely transmitted to the downstream (downstream application) to ensure that the changes in the table structure and data in the lake can be timely transmitted to the downstream, so as to ensure the consistency and correctness of the data of the entire data link.

[0127] In this scenario example, in order to ensure the latest data in the lake table, the arrival of the interface file server on the file transfer server is automatically detected, and the latest files are automatically batch loaded each time to put the file data information into the lake. The batch partitions loaded each time are kept consistent with the interface file date. The specific processing flow may include the following:

[0128] 1) Perform pre-processing check on incoming files before loading. This includes checking the length, code system, name, transmission information, structure, arrival status and other information of the file. The checking process can specifically include: checking whether the field length meets the requirements; whether the field format is a fixed length; whether the field separator is correct; if any of the above tests are inconsistent, the batch loading process will fail, prompting the loader to interrupt, etc. This is a check before loading.

[0129] 2) Automatically load the latest files into the data lake table and keep them consistent with the upstream files; at the same time, the batch loading process will perform data quality checks on the files entering the lake, such as error row fault tolerance and file primary key. The above loading method is called the source-attached loading method, which belongs to the post-loading check.

[0130] 3) Call the blood relationship through the data lake table; and according to the blood relationship, associate the downstream job scheduling that uses the data lake table, and automatically start the full process job of the caller application to pass on the latest data in the lake. Among them, when this operation step detects that there is a data update in the data lake, it will determine the downstream application associated with the updated data according to the blood relationship, and then automatically pass the updated data to the downstream application for use. For example, the account deposit and withdrawal transaction data is automatically loaded into the data lake, and the corresponding downstream is started to perform data report calculation tasks such as summarizing the account transaction amount of the day and associating the account transaction relationship. The above-mentioned blood relationship can specifically refer to the upstream and downstream program dependency relationship.

[0131] 4) If the file is not updated, it will not be loaded, and the error message will be notified (by email, etc.) to each application for synchronization modification.

[0132] For details, please refer to Figure 4 shown.

[0133] In this scenario example, during the automated test design process, the data lake scheduling server mounts the server where the upstream file is downloaded and stored. After each interface file of the upstream file is automatically loaded and the date of the upstream file download is recorded, when the upstream file is downloaded again, the tool will automatically compare it with the download date of the last file. If the latest file is later than the last loaded file, the tool will reload the downloaded file, otherwise it will not be loaded.

[0134] Before loading the interface file, the data lake will perform a preprocessing check on the file, including the size, name, storage path, file length, encoding, structural information, storage format of the loaded file, whether the dependent pre-jobs have been completed, etc. The check method is to compare one by one with the data lake table configuration method. Automatic loading will be performed after the preprocessing check passes. Since the test files will change and download frequently during the test process, and the frequency of automatic interface file loading is generally once a day, in the design of the automation tool, the file preprocessing check and interface file batch loading can be separated as an independent automated test function for testers to load at any time according to their own needs. Testers only need to enter the relevant test environment, loading job / file name, and loading date in the automated test interface.

[0135] In this scenario example, after the data lake table is automatically loaded successfully, the interface file source will be added to the data lake table. If the loading fails, the failure reason will be notified through detailed log records. The log information will include error information of each check item of the file preprocessing upstream interface file. For details, please refer to Figure 5 shown.

[0136] In this scenario example, the data stored in the data lake table includes basic user information, bank account transaction information, bank institution information, etc. After the data is updated, the automated testing tool associates the call relationship of the data lake table, and associates the related jobs of each downstream application that uses the table according to the blood relationship diagram. By automatically scheduling the batches of each downstream application, the data in the data lake table is transmitted to each application. If there is a scheduling job failure, it will automatically notify each downstream application and send log information at the same time. For details, please refer to Figure 6 and Figure 7 .

[0137] In this scenario example, the automatic batch loading test of the interface file can specifically refer to the real-time and effective transmission of the data of the entire link of interface file-data lake table-data lake table user in an automated way to ensure the validity of the data during the entire test process. Figure 8 shown.

[0138] Through the above scenario examples, it is verified that the data processing method provided by the embodiment of this specification realizes the data linkage from the interface file to the basic data of the data lake table and the entire data chain of the data lake data user through automatic detection and real-time batch loading, and transmits the data quickly and effectively to ensure the freshness and effectiveness of the data. Specifically, it has the following advantages: 1. High data timeliness. After the source data of the data lake table is updated, the data is passed to the data lake as soon as possible, and at the same time, the data lake data is passed to the data user through the blood relationship chain through the whole process scheduling; 2. The whole link of data structure changes is synchronized in time. When the source data structure and other information change, the various users on the data link are notified through the blood relationship to synchronize the table structure update.

[0139] Although the present specification provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.

[0140] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0141] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer-readable storage media including storage devices.

[0142] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that the present specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can essentially be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0143] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0144] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A data processing method, characterized in that: Applicable to data lake servers, including: Detect whether the target data downloaded by the upstream application is received; wherein, when the upstream application downloads the target data, it generates a corresponding timestamp according to the download time and adds the timestamp to the target data; When it is determined that target data transmitted by the upstream application is received, pre-processing and checking the target data; When it is determined that the target data preprocessing check has passed, the target data is loaded into the target data lake in a source-to-lake manner; wherein the target data lake is used to store data shared by upstream applications and downstream applications; the target data lake also stores a lineage relationship table, which records the lineage relationships between multiple upstream applications and downstream applications connected to the target data lake; Obtaining and determining downstream applications associated with the target data based on the lineage relationship of the target data lake; including: searching the lineage relationship table of the target data lake based on the application identifier of the upstream application, obtaining the lineage relationship including the application identifier of the upstream application; determining downstream applications associated with the target data based on the lineage relationship; Calling the downstream application to batch load and / or process the target data; Among them, after determining that the target data downloaded by the upstream application is received, the method also includes: determining the download time of the target data according to the timestamp carried in the target data; obtaining the download time of the target data downloaded by the upstream application with the same application identifier as the last received, as a reference time; comparing the reference time with the download time of the target data, and when it is determined that the reference time is later than the download time of the target data, determining that the target data is expired invalid data, and refusing to load the target data into the target data lake for storage; on the contrary, when it is determined that the reference time is earlier than the download time of the target data, determining that the target data is valid data that has not expired, and continuing subsequent data processing.

2. The method according to claim 1, characterized in that Performing a pre-processing check on the target data, including: According to the preset data inspection rules, the target characteristics of the target data are inspected to determine whether the target characteristics meet the preset requirements; wherein the target characteristics include at least one of the following: the length of the data, the code system of the data, the name of the data, the transmission information of the data, the structure of the data, the storage path of the data, and the storage format of the data; When it is determined that the target features meet the preset requirements, it is determined that the target data preprocessing check has passed.

3. The method according to claim 1, characterized in that: The target data is loaded into the target data lake in a source-to-lake manner, including: Get data quality check rules that match the upstream application; While the target data is loaded into the target data lake, key data is extracted from the target data entering the lake according to the data quality check rules, and a data quality check is performed on the key data.

4. The method according to claim 1, characterized in that: After calling the downstream application to batch load and / or process the target data, the method further includes: Determining whether a processing result of calling the downstream application to batch load and / or process the target data is a failure; When the processing result is determined to be a failure, an error log is generated; Sending the error log to the downstream application.

5. The method according to claim 1, characterized in that The target data includes an interface file.

6. The method according to claim 5, characterized in that The upstream application includes a bank card application; the downstream application includes an account risk control application.

7. A data processing device, characterized in that: include: A detection module, used to detect whether the target data downloaded by the upstream application is received; wherein, when the upstream application downloads the target data, it generates a corresponding timestamp according to the download time, and adds the timestamp to the target data; A checking module, used for performing a pre-processing check on the target data when determining that the target data is received and downloaded by the upstream application; A loading module is used to load the target data into a target data lake in a source-to-lake manner when it is determined that the target data preprocessing check has passed; wherein the target data lake is used to store data shared by upstream applications and downstream applications; the target data lake also stores a lineage relationship table, which records the lineage relationships between multiple upstream applications and downstream applications connected to the target data lake; A determination module is used to obtain and determine the downstream application associated with the target data according to the blood relationship of the target data lake; the determination module is used to retrieve the blood relationship table of the target data lake according to the application identifier of the upstream application, obtain the blood relationship including the application identifier of the upstream application; and determine the downstream application associated with the target data according to the blood relationship; A calling module, used for calling the downstream application to batch load and / or process the target data; Among them, after determining that the target data downloaded by the upstream application is received, the device is also used to determine the download time of the target data according to the timestamp carried in the target data; obtain the download time of the target data downloaded by the upstream application with the same application identifier as the last received, as the reference time; compare the reference time with the download time of the target data, and when it is determined that the reference time is later than the download time of the target data, determine that the target data is expired invalid data, and refuse to load the target data into the target data lake for storage; on the contrary, when it is determined that the reference time is earlier than the download time of the target data, determine that the target data is valid data that has not expired, and continue subsequent data processing.

8. A server, characterized in that: The method comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the instructions.

9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a computer device, the steps of the method described in any one of claims 1 to 6 are implemented.

Images