Method, system, device and storage medium for processing data

Abstract

The embodiments of the present application disclose a method, system, device and storage medium for processing data. The method is applied to a data warehouse, which includes a data preparation layer, a detail data layer, a data service layer and an application data service layer in sequence. By triggering a crawler to capture metadata from a storage device and triggering the detail data layer, the data service layer and the application data service layer to perform operations, the construction of a data warehouse is realized, and the workload of developers in maintaining the compatibility between various components of the data warehouse and the underlying services is reduced, so that developers can concentrate on the development of business indicators and reduce the construction and maintenance costs of the data warehouse.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a method for processing data, a system for processing data, a device for processing data, and a machine-readable storage medium. Background Art

[0002] Enterprises usually build data warehouses based on native Hadoop, highly integrated Hadoop distributions (Cloudera's Distribution Including Apache Hadoop, CDH), MaxCompute, or Dataworks based on distributed system infrastructure.

[0003] The Hadoop-based data warehouse is relatively open and suitable for giant companies with large technical teams to make some underlying modifications and development for their own businesses. However, the maintenance cost of the data warehouse is very high and requires a professional operation and maintenance team, which is not suitable for small and medium-sized start-ups. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, system, device and storage medium for processing data, aiming to solve the problem of high cost of building data warehouses for enterprises in the prior art.

[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides a method for processing data, which is applied to a data warehouse, wherein the data warehouse sequentially includes a data preparation layer, a detail data layer, a data service layer, and an application data service layer, and the method includes:

[0006] In the data preparation layer, the crawler is triggered to fetch metadata of the raw data stored in the storage device;

[0007] At the detail data layer, monitor whether the crawling of the data preparation layer is successful. If the crawling of the data preparation layer is successful, trigger this layer to execute the job based on the metadata captured by the data preparation layer. After obtaining the execution result of this layer, output it to the storage device, and monitor whether the execution of the job of this layer is successful. If the execution of the job of this layer is successful, trigger the crawler to crawl the metadata of the execution result of this layer in the storage device;

[0008] At the data service layer, monitor whether the crawling of the detail data layer is successful. If the crawling of the detail data layer is successful, trigger this layer to execute the job based on the metadata captured by the detail data layer. After obtaining the execution result of this layer, output it to the storage device, and monitor whether the execution of the job of this layer is successful. If the execution of the job of this layer is successful, trigger the crawler to crawl the metadata of the execution result of this layer in the storage device;

[0009] In the application data service layer, monitor whether the crawling of the data service layer is successful. If the crawling of the data service layer is successful, trigger this layer to execute the job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data.

[0010] In an embodiment of the present application, triggering this layer to execute a job based on metadata captured by the data preparation layer, and outputting the execution result of this layer to a storage device after obtaining the execution result of this layer, includes:

[0011] Trigger the execution of a job to clean the metadata captured by the data preparation layer to obtain cleaned data;

[0012] Output the cleaned data to the storage device.

[0013] In an embodiment of the present application, triggering this layer to execute a job based on metadata captured by the detail data layer, and outputting the execution result of this layer to a storage device after obtaining the execution result of this layer, includes:

[0014] Trigger the execution of a job that aggregates the metadata captured at the detail data layer to obtain a wide table;

[0015] Output a wide table to a storage device.

[0016] In the embodiment of the present application, the storage device is a storage unit S3 of the cloud computing service AWS.

[0017] In an embodiment of the present application, the method for processing data further includes:

[0018] The business indicator data is output to the storage unit S3 and the business database.

[0019] In an embodiment of the present application, the method for processing data further includes:

[0020] At the application data service layer, monitor whether the job execution at this layer is successful;

[0021] If the job execution at this layer is successful, the crawler is triggered to fetch the metadata of the business indicator data in the storage unit S3;

[0022] Output the metadata of business indicator data to the business database.

[0023] In an embodiment of the present application, the method for processing data further includes:

[0024] In the data preparation layer, the first crawler is triggered by a timer trigger to crawl the metadata of the original data;

[0025] At the detail data layer, the first trigger is used to monitor whether the crawling of the data preparation layer is successful. If the crawling of the data preparation layer is successful, the current layer is triggered to execute the job based on the metadata captured by the data preparation layer. After obtaining the execution result of the current layer, it is output to the storage device. The second trigger is used to monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, the second crawler is triggered to crawl the metadata of the execution result of the current layer in the storage device.

[0026] At the data service layer, the third trigger is used to monitor whether the crawling of the detail data layer is successful. If the crawling of the detail data layer is successful, the current layer is triggered to execute the job based on the metadata crawled by the detail data layer, and the execution result of the current layer is output to the storage device after the execution result of the current layer is obtained. The fourth trigger is used to monitor whether the execution of the current layer is successful. If the execution of the current layer is successful, the third crawler is triggered to crawl the metadata of the execution result of the current layer in the storage device;

[0027] In the application data service layer, the fifth trigger is used to monitor whether the crawling of the data service layer is successful. If the crawling of the data service layer is successful, this layer is triggered to execute the job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data.

[0028] In an embodiment of the present application, the method for processing data further includes:

[0029] Create workflows for data warehouses;

[0030] Determine the color of the node corresponding to the crawler in the workflow;

[0031] When the color is the first preset color, it is determined that the crawler has successfully captured the image;

[0032] When the color is the second preset color, it is determined that the crawler fails to crawl.

[0033] A second aspect of the present application provides a device for processing data, characterized in that it is applied to a data warehouse, the data warehouse sequentially includes a data preparation layer, a detail data layer, a data service layer and an application data service layer, and the device includes:

[0034] A first data processing module is used to trigger a crawler to crawl metadata of raw data stored in a storage device at a data preparation layer;

[0035] The second data processing module is used to monitor whether the crawling of the data preparation layer is successful at the detail data layer. If the crawling of the data preparation layer is successful, the current layer is triggered to execute the job based on the metadata crawled by the data preparation layer, and after obtaining the execution result of the current layer, it is output to the storage device, and monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, the crawler is triggered to crawl the metadata of the execution result of the current layer in the storage device;

[0036] The third data processing module is used to monitor whether the crawling of the detail data layer is successful at the data service layer. If the crawling of the detail data layer is successful, trigger the current layer to execute the job based on the metadata crawled by the detail data layer, output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, trigger the crawler to crawl the metadata of the execution result of the current layer in the storage device;

[0037] The fourth data processing module is used to monitor whether the data service layer's crawling is successful in the application data service layer. If the data service layer crawls successfully, it triggers this layer to execute a job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data.

[0038] A third aspect of the present application provides a device for processing data, including:

[0039] a memory for storing programs for processing data;

[0040] The processor is configured to call the program for processing data from the memory so that the processor can execute the above method for processing data.

[0041] A fourth aspect of the present application provides a system for processing data, comprising:

[0042] Storage device, used to store original data;

[0043] The above-mentioned device for processing data.

[0044] In the embodiment of the present application, the storage device is a storage unit S3 of AWS, and the system further includes:

[0045] AWS cloud server, used to receive raw data;

[0046] The storage unit S3 is used to obtain the original data from the AWS cloud server and store the original data.

[0047] A fifth aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configure the processor to execute the above-mentioned method for processing data.

[0048] Through the above technical solution, by triggering the crawler to capture metadata from the storage device, and triggering the detail data layer, data service layer and application data service layer to perform jobs, the construction of the data warehouse is realized, reducing the workload of developers to maintain the compatibility between the various components of the data warehouse and the underlying services, so that developers can concentrate on the development of business indicators, reducing the construction and maintenance costs of the data warehouse.

[0049] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0051] Figure 1 A schematic diagram of an application environment of a method for processing data according to an embodiment of the present application is schematically shown;

[0052] Figure 2 A schematic diagram of a process for processing data according to an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of a process flow of a method for processing data according to another embodiment of the present application is shown;

[0054] Figure 4 A flowchart of another embodiment of the present application is schematically shown;

[0055] Figure 5 The workflow diagram of the embodiment of the present application is schematically shown;

[0056] Figure 6 A functional module diagram of a system for processing data according to an embodiment of the present application is schematically shown;

[0057] Figure 7 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0060] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0061] The method for processing data provided in this application can be applied to Figure 1 In the application environment shown. Among them, the device realizes information interaction with the cloud service through the Internet of Things technology. The cloud service is the logic developed by the back-end developers using code. For example, the device sends order information to the cloud, and the cloud sends the information of starting production to the device. The cloud service sends all business data to the business database for storage, and sends part of the data to the storage device for analysis by the big data engineer who builds the data warehouse. The big data engineer captures the data of the storage device for layer-by-layer analysis, and stores the final indicators in the business database. In an example, the Internet of Things technology and cloud service can be built based on the cloud computing service AWS, the cloud service is deployed on the cloud server of AWS, and the business database is the MongoDB database. The cloud server of AWS sends all business data to MongoDB for storage, and sends part of the data to the storage unit S3 of AWS for analysis by the big data engineer who builds the data warehouse. The big data engineer captures the data of S3 for analysis, and stores the final business indicator data in MongoDB.

[0062] Figure 2 The flowchart of the method for processing data according to an embodiment of the present application is schematically shown. Figure 2 As shown, in one embodiment of the present application, a method for processing data is provided. This embodiment mainly applies the method to the above Figure 1The data warehouse in the embodiment of the present invention comprises a data preparation layer, a detail data layer, a data service layer and an application data service layer in sequence. Specifically, the data preparation layer can be referred to as an ODS (Operation Data Store) layer, the detail data layer can be referred to as a DWD (Data Warehouse Details) layer, the data service layer can be referred to as a DWS (Data Warehouse Service) layer and the application data service layer can be referred to as an ADS (Application Data Service) layer. The method may include the following steps:

[0063] S10: At the data preparation layer, the crawler is triggered to capture metadata of the original data stored in the storage device.

[0064] It should be noted that data warehouse and database are two different concepts. Databases usually include relational databases and non-relational databases (such as MongDB database, which is a non-relational database). They are mainly used to record transactions. When the data in the database accumulates to a certain extent, the concept of data warehouse is born. Data warehouse focuses more on data analysis and gives indicators and conclusions. For example, the sales orders that occur every day are stored in the database. The database only records the facts of these transactions. If you need to determine which time period of the day has the most sales orders, or the age distribution of the buyers of sales orders in each time period, you need a data warehouse to analyze the existing data.

[0065] Regarding the architecture of the data warehouse, a four-layer model is adopted in the embodiment of the present application, which is the data preparation layer (ODS layer), the detail data layer (DWD layer), the data service layer (DWS layer) and the application data service layer (ADS layer). For example, the ODS layer is used to store the original data, the DWD layer is used to parse the original data, clean the dirty data, desensitize, etc., the DWS layer is used to aggregate the information of the order into a wide table with a relatively large number of fields, and the ADS layer is used to output specific business indicator data.

[0066] It should be understood that the original data includes business data stored by cloud servers in storage devices, such as order sales data, promotional activity data, user authorization data, device information data, etc.

[0067] The storage device can be the storage unit S3 of the cloud computing service (Amazon Web Services, AWS). The enterprise's data warehouse is built through AWS. The Glue job in the DWD layer mainly includes the logic of data cleaning, and the Glue job in the DWS layer and ADS layer mainly reflects the logic of batch processing business.

[0068] Since the project has limited human resources and a short delivery cycle, AWS Glue can be used to quickly build a lightweight enterprise data warehouse. Developers do not need to maintain the compatibility between components and the underlying services, and can focus on the development of business indicators. For small-scale enterprises, using AWS Glue to process micro-batch data saves both time and money in running jobs. Since the analyzed data occasionally involves ad hoc query scenarios, the natural integration of AWS Glue and AWS Athena (an AWS ad hoc query service) also facilitates use in the above scenarios.

[0069] In the specific implementation, the crawler can be triggered by the corresponding trigger. In the ODS layer, the first trigger can be timed, and other triggers can trigger the next task by monitoring whether the previous task is successfully executed. If the execution is successful, it will be triggered, and if the execution fails, it will be terminated. Each crawler must specify the crawling location when it is created. Since all the data to be parsed can be stored in S3, each crawler must be configured with the corresponding S3 path when it is created.

[0070] Taking the original data as order sales data as an example, when an order occurs, there will be some uplink and downlink messages between the device and the cloud service. The data of each order will be sent to the business database MongoDB and S3 respectively by the cloud. The crawler will crawl the S3 data for analysis and store it in S3 after analysis. When the ADS layer completes the analysis, the business indicator data can also be stored in S3 and MongoDB at the same time. The data in MongoDB can be used by front-end developers for data display.

[0071] At the ODS layer, a timer trigger can trigger the execution of the entire workflow at 12:15 a.m. every day, and trigger the crawler to crawl the metadata of the original data of the storage device on the previous day.

[0072] S20: At the detail data layer, monitor whether the crawling of the data preparation layer is successful. If the crawling of the data preparation layer is successful, trigger this layer to execute the job based on the metadata crawled by the data preparation layer. After obtaining the execution result of this layer, output it to the storage device, and monitor whether the execution of the job of this layer is successful. If the execution of the job of this layer is successful, trigger the crawler to crawl the metadata of the execution result of this layer in the storage device.

[0073] It should be understood that the DWD layer mainly parses the metadata of the original data, removes dirty data, desensitizes, etc. Specifically, after the crawler in the ODS layer successfully captures, the DWD layer job is triggered by the trigger to execute the job, and the metadata captured by the ODS layer is cleaned to obtain the cleaned data (i.e., the DWD layer table), and the cleaned data is output to the storage device.

[0074] After the job is executed, the next level of crawler is triggered based on whether the job is executed successfully to fetch the metadata of the DWD layer table for light aggregation at the DWS layer to form a wide table.

[0075] In the workflow of the data warehouse, two types of triggers can be set. One is a timed trigger, which is used at the starting node of each workflow to start executing tasks at a preset time. The other is a conditional trigger, which triggers the execution of the next task by monitoring whether the previous task is successful. If successful, it will be executed, and if failed, it will be terminated. There are two types of conditional triggers, one is the any type and the other is the all type. The any type means that as long as one of the monitored tasks is successful, the next task can be run; the all type means that all monitored tasks must be successfully executed before the next task can be run.

[0076] Take the data warehouse workflow of the platform tenant, where the business indicator data is the number of orders, cups, and amount sold every day as an example: At the DWD layer, after the conditional trigger monitors the successful execution of the previous crawler, it triggers the job, performs preliminary data cleaning on the metadata of the original data, obtains the execution result (i.e., the DWD layer table), and outputs S3. After the conditional trigger monitors the successful execution of the previous job, it triggers the crawler to capture the metadata of the DWD layer table.

[0077] S30: At the data service layer, monitor whether the crawling of the detail data layer is successful. If the crawling of the detail data layer is successful, trigger this layer to execute the job based on the metadata crawled by the detail data layer. After obtaining the execution result of this layer, output it to the storage device, and monitor whether the execution of the job of this layer is successful. If the execution of the job of this layer is successful, trigger the crawler to crawl the metadata of the execution result of this layer in the storage device.

[0078] It should be understood that at the DWS layer, order information is mainly aggregated into a wide table with a relatively large number of fields, so as to be used to calculate specific business indicator data based on this wide table at the ADS layer.

[0079] Take the data warehouse workflow of the platform tenants, where the business indicator data is the number of orders, cups, and amount sold every day as an example: the trigger listens to the crawler in the DWD layer successfully grabbing the metadata of the DWD layer table, triggering the job execution of the DWS layer to aggregate the metadata grabbed by the DWD layer to obtain a wide table, which only retains the record of the final status of each order and outputs the wide table to S3.

[0080] After the trigger monitors that the job at the DWS layer is successfully executed, it triggers the crawler to crawl the metadata of the enriched table.

[0081] S40: In the application data service layer, monitor whether the crawling of the data service layer is successful. If the crawling of the data service layer is successful, trigger this layer to execute a job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data.

[0082] In the specific implementation, at the ADS layer, it is also possible to monitor whether the job execution at this layer is successful; if the job execution at this layer is successful, the crawler is triggered to capture the metadata of the business indicator data in the storage device; and the metadata of the business indicator data is output to the business database.

[0083] It should be understood that the ADS layer is mainly used to produce specific business indicator data. Take the data warehouse workflow of the platform tenant, where the business indicator data is the number of orders, cups, and amount sold every day as an example: in the ADS layer, after the trigger monitors the successful execution of the crawler in the DWS layer, it triggers the job execution of the ADS layer, and calculates the number of orders, cups, and amount sold by the tenant the previous day through the code, and outputs them to S3 and MongoDB at the same time. After the trigger monitors the successful execution of the previous job, it triggers the crawler to capture the metadata of the ADS layer table and finally outputs it to MongoDB.

[0084] It should be noted that each crawler or job requires a trigger to trigger, and a trigger can trigger multiple crawlers or multiple jobs. There is no special corresponding relationship between triggers, crawlers and jobs, and the embodiments of the present application do not limit this.

[0085] The embodiment of the present application realizes the construction of a data warehouse by triggering a crawler to capture metadata from a storage device and triggering the DWD layer, DWS layer and ADS layer to perform jobs, thereby reducing the workload of developers in maintaining the compatibility between various components of the data warehouse and the underlying services, allowing developers to concentrate on the development of business indicators and reducing the construction and maintenance costs of the data warehouse.

[0086] Figure 3 The flowchart of the method for processing data according to another embodiment of the present application is schematically shown. Figure 3 and Figure 4 The method may further include the following steps:

[0087] S10': In the data preparation layer, the first crawler is triggered by a timing trigger to crawl metadata of the original data.

[0088] S20': At the detail data layer, monitor whether the crawling of the data preparation layer is successful through the first trigger. If the crawling of the data preparation layer is successful, trigger this layer to execute the job based on the metadata crawled by the data preparation layer, and output the execution result of this layer to the storage device after obtaining it. Monitor whether the execution of the job of this layer is successful through the second trigger. If the execution of the job of this layer is successful, trigger the second crawler to crawl the metadata of the execution result of this layer in the storage device.

[0089] S30': In the data service layer, monitor whether the crawling of the detail data layer is successful through the third trigger. If the crawling of the detail data layer is successful, trigger this layer to execute the job based on the metadata crawled by the detail data layer, and output the execution result of this layer to the storage device after obtaining it. Monitor whether the execution of the job of this layer is successful through the fourth trigger. If the execution of the job of this layer is successful, trigger the third crawler to crawl the metadata of the execution result of this layer in the storage device.

[0090] S40': In the ADS layer, the fifth trigger is used to monitor whether the crawling of the data service layer is successful. If the crawling of the data service layer is successful, this layer is triggered to execute the job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data.

[0091] It should be understood that in order to facilitate the maintenance of the workflow and make the processing logic clearer, triggers can be set for each layer, and different triggers can trigger different jobs or crawlers. In an embodiment of the present application, at the ODS layer, the first crawler Crawler01 is triggered by a timing trigger; at the DWD layer, the DWD job is triggered by the first trigger Trigger01, and the second crawler Crawler02 is triggered by the second trigger Trigger02; at the DWS layer, the DWSjob is triggered by the third trigger Trigger03, and the third crawler Crawler03 is triggered by the fourth trigger Trigger04; at the ADS layer, the ADS job is triggered by the fifth trigger Trigger05. When there are 3 business indicators, the ADS jobs are ADS job01, ADS job02 and ADSjob03 respectively. More triggers (such as Trigger06, Trigger07 and Trigger08) can also be set at the ADS layer to trigger crawlers (Crawler04, Crawler05 and Crawler06) to crawl data.

[0092] refer to Figure 5There are many ways to determine whether the crawler has successfully crawled. In order to determine the judgment result more intuitively, the following method can be used for judgment: create a workflow for the data warehouse; determine the color of the node corresponding to the crawler in the workflow; when the color is the first preset color, determine that the crawler has successfully crawled; when the color is the second preset color, determine that the crawler has failed.

[0093] In one example, when the color of a node in the workflow is displayed as green, it is determined that the crawler crawls successfully, and when the color is displayed as red, it is determined that the crawler crawls failed.

[0094] The embodiment of the present application sets triggers at the ODS layer, DWD layer, DWS layer and ADS layer respectively, and different crawlers and jobs are triggered by different triggers, so that the maintenance of the workflow is more convenient and the processing logic is clearer.

[0095] Figure 2 and Figure 3 FIG. 1 is a flow chart of a method for processing data in one embodiment. It should be understood that although Figure 2 and Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0096] In one embodiment, Figure 6 As shown, a device for processing data is provided, which is applied to a data warehouse. The data warehouse includes a data preparation layer, a detail data layer, a data service layer, and an application data service layer in sequence. The device includes:

[0097] The first data processing module 10 is used to trigger the crawler to crawl metadata of the original data stored in the storage device at the data preparation layer;

[0098] The second data processing module 20 is used to monitor whether the crawling of the data preparation layer is successful at the detail data layer. If the crawling of the data preparation layer is successful, trigger the current layer to execute the job based on the metadata crawled by the data preparation layer, output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, trigger the crawler to crawl the metadata of the execution result of the current layer in the storage device;

[0099] The third data processing module 30 is used to monitor whether the crawling of the detail data layer is successful at the data service layer. If the crawling of the detail data layer is successful, trigger the current layer to execute the job based on the metadata crawled by the detail data layer, output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, trigger the crawler to crawl the metadata of the execution result of the current layer in the storage device;

[0100] The fourth data processing module 40 is used to monitor whether the data service layer is successfully captured in the application data service layer. If the data service layer is successfully captured, it triggers this layer to execute a metadata job based on the execution result of the data service layer to obtain business indicator data corresponding to the original data.

[0101] Furthermore, the second data processing module 20 is also used to trigger the execution of a job for cleaning the metadata captured by the data preparation layer to obtain cleaned data; and output the cleaned data to a storage device.

[0102] Furthermore, the third data processing module 30 is also used to trigger the execution of a job of aggregating the metadata captured from the detail data layer to obtain a wide table; and output the wide table to a storage device.

[0103] Furthermore, the storage device is a storage unit S3 of the cloud computing service AWS.

[0104] Furthermore, the fourth data processing module 40 is also used to output the business indicator data to the storage unit S3 and the business database.

[0105] Furthermore, the fourth data processing module 40 is also used to monitor whether the job execution at this layer is successful in the application data service layer; if the job execution at this layer is successful, trigger the crawler to capture the metadata of the business indicator data in the storage unit S3; and output the metadata of the business indicator data to the business database.

[0106] Furthermore, the first data processing module 10 is also used to trigger the first crawler to crawl metadata of the original data through a timing trigger at the data preparation layer;

[0107] The second data processing module 20 is also used to monitor whether the crawling of the data preparation layer is successful through the first trigger at the detail data layer, and if the crawling of the data preparation layer is successful, trigger the current layer to execute the job based on the metadata crawled by the data preparation layer, and output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful through the second trigger, and if the execution of the job of the current layer is successful, trigger the second crawler to crawl the metadata of the execution result of the current layer in the storage device;

[0108] The third data processing module 30 is also used to monitor whether the crawling of the detail data layer is successful through the third trigger at the data service layer. If the crawling of the detail data layer is successful, trigger the current layer to execute the job based on the metadata crawled by the detail data layer, and output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful through the fourth trigger. If the execution of the job of the current layer is successful, trigger the third crawler to crawl the metadata of the execution result of the current layer in the storage device;

[0109] The fourth data processing module 40 is also used to monitor whether the data service layer's crawling is successful through the fifth trigger in the application data service layer. If the data service layer crawls successfully, it triggers this layer to execute a job based on the metadata of the execution result of the data service layer to obtain business indicator data corresponding to the original data.

[0110] Furthermore, the device for processing data also includes a crawling result judgment module, which is used to create a workflow for the data warehouse; determine the color of the node corresponding to the crawler in the workflow; when the color is a first preset color, determine that the crawler has successfully crawled; when the color is a second preset color, determine that the crawler has failed.

[0111] An embodiment of the present application also provides a device for processing data, including a processor and a memory, the memory being used to store a program for processing data; the processor being configured to call the program for processing data from the memory so that the processor can execute the above-mentioned method for processing data.

[0112] An embodiment of the present application also provides a system for processing data, including: a storage device for storing original data; and the above-mentioned device for processing data.

[0113] In one example, the storage device is a storage unit S3 of AWS, and the system further includes: an AWS cloud server, used to receive original data; and a storage unit S3, used to obtain the original data from the AWS cloud server and store the original data.

[0114] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor is configured to execute the above-mentioned method for processing data.

[0115] The embodiment of the present application provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a method for processing data is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0116] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0122] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0123] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0125] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for processing data, characterized in that Applied to a data warehouse, the data warehouse sequentially comprises a data preparation layer, a detail data layer, a data service layer and an application data service layer, the method comprises: In the data preparation layer, the crawler is triggered to fetch metadata of the raw data stored in the storage device; At the detail data layer, monitor whether the crawling of the data preparation layer is successful. If the crawling of the data preparation layer is successful, trigger the current layer to perform a cleaning operation based on the metadata captured by the data preparation layer, output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the current layer is successful. If the execution of the current layer is successful, trigger the crawler to crawl the metadata of the execution result of the current layer in the storage device, and the storage device is the storage unit S3 of the cloud computing service AWS; At the data service layer, monitor whether the crawling of the detail data layer is successful. If the crawling of the detail data layer is successful, trigger the current layer to perform the aggregation job based on the metadata crawled by the detail data layer, output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, trigger the crawler to crawl the metadata of the execution result of the current layer in the storage device; At the application data service layer, monitor whether the crawling of the data service layer is successful. If the crawling of the data service layer is successful, trigger this layer to execute the job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data, and output the business indicator data to the storage unit S3 and the business database. Monitor whether the execution of the job of this layer is successful. If the execution of the job of this layer is successful, trigger the crawler to crawl the metadata of the business indicator data in the storage unit S3, and output the metadata of the business indicator data to the business database.

2. The method according to claim 1, characterized in that The triggering of the current layer to perform a cleaning operation based on the metadata captured by the data preparation layer, and outputting the execution result of the current layer to the storage device after obtaining the execution result of the current layer, includes: Triggering execution of a job for cleaning the metadata captured by the data preparation layer to obtain cleaned data; The cleaned data is output to the storage device.

3. The method according to claim 2, characterized in that The triggering of the current layer to perform the aggregation operation based on the metadata captured by the detail data layer, and outputting the execution result of the current layer to the storage device after obtaining the execution result of the current layer, includes: triggering execution of a job for aggregating metadata captured from the detail data layer to obtain a wide table; The wide table is output to the storage device.

4. The method according to claim 1, characterized in that: The method further comprises: At the data preparation layer, triggering the first crawler to crawl the metadata of the original data through a timing trigger; At the detail data layer, a first trigger is used to monitor whether the crawling of the data preparation layer is successful. If the crawling of the data preparation layer is successful, the current layer is triggered to execute a job based on the metadata crawled by the data preparation layer, and the execution result of the current layer is output to the storage device after the execution result of the current layer is obtained. A second trigger is used to monitor whether the execution of the current layer is successful. If the execution of the current layer is successful, the second crawler is triggered to crawl the metadata of the execution result of the current layer in the storage device. At the data service layer, a third trigger is used to monitor whether the crawling of the detail data layer is successful. If the crawling of the detail data layer is successful, the current layer is triggered to execute a job based on the metadata crawled by the detail data layer, and the execution result of the current layer is output to the storage device after the execution result of the current layer is obtained. A fourth trigger is used to monitor whether the execution of the current layer is successful. If the execution of the current layer is successful, the third crawler is triggered to crawl the metadata of the execution result of the current layer in the storage device. In the application data service layer, the fifth trigger is used to monitor whether the crawling of the data service layer is successful. If the crawling of the data service layer is successful, this layer is triggered to execute the job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data.

5. The method according to claim 1, characterized in that Also includes: Creating a workflow for the data warehouse; Determine the color of the node corresponding to the crawler in the workflow; When the color is the first preset color, determining that the crawler has successfully captured; When the color is the second preset color, it is determined that the crawler fails to crawl.

6. A device for processing data, characterized in that Applied to a data warehouse, the data warehouse sequentially comprises a data preparation layer, a detail data layer, a data service layer and an application data service layer, the device comprises: A first data processing module is used to trigger a crawler to crawl metadata of raw data stored in a storage device at a data preparation layer; The second data processing module is used to monitor whether the crawling of the data preparation layer is successful at the detail data layer. If the crawling of the data preparation layer is successful, the current layer is triggered to perform a cleaning operation based on the metadata captured by the data preparation layer, and after obtaining the execution result of the current layer, the module outputs the result to the storage device, and monitors whether the execution of the current layer is successful. If the execution of the current layer is successful, the crawler is triggered to crawl the metadata of the execution result of the current layer in the storage device, and the storage device is the storage unit S3 of the cloud computing service AWS; The third data processing module is used to monitor whether the crawling of the detail data layer is successful at the data service layer. If the crawling of the detail data layer is successful, trigger the current layer to perform an aggregation job based on the metadata crawled by the detail data layer, output the execution result of the current layer to the storage device after obtaining it, and monitor whether the execution of the job of the current layer is successful. If the execution of the job of the current layer is successful, trigger the crawler to crawl the metadata of the execution result of the current layer in the storage device; The fourth data processing module is used to monitor whether the crawling of the data service layer is successful in the application data service layer. If the crawling of the data service layer is successful, trigger this layer to execute the job based on the metadata of the execution result of the data service layer to obtain the business indicator data corresponding to the original data, and output the business indicator data to the storage unit S3 and the business database. Monitor whether the execution of the job of this layer is successful. If the execution of the job of this layer is successful, trigger the crawler to crawl the metadata of the business indicator data in the storage unit S3, and output the metadata of the business indicator data to the business database.

7. A device for processing data, characterized in that include: a memory for storing programs for processing data; A processor is configured to call the program for processing data from the memory so that the processor can execute the method for processing data according to any one of claims 1 to 5.

8. A system for processing data, characterized in that include: A storage device for storing raw data; The device for processing data according to claim 7.

9. The system according to claim 8, characterized in that The storage device is a storage unit S3 of AWS, and the system further includes: AWS cloud server, used to receive the original data; The storage unit S3 is used to obtain the original data from the AWS cloud server and store the original data.

10. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to perform the method for processing data according to any one of claims 1 to 5.

Images