Data processing method, device, electronic device and computer readable medium

Abstract

The embodiments of the present application provide a data processing method, device, electronic device and computer-readable medium, which relate to the field of data warehouse. Among them, the method includes: uniformly parsing the business object data obtained in real time from different data sources to obtain business object structured data for generating business object data tables; loading the business object structured data into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data. By uniformly parsing the business object data obtained in real time from different data sources, the embodiments of the present application only need to develop, run and maintain one set of ETL parsing logic, without developing, running and maintaining multiple sets of ETL parsing logic, which greatly reduces labor costs.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of data warehouses, and in particular, to a data processing method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, the data warehouse processes the data in the business system based on the ETL data processing theory. ETL refers to Extraction, Transformation, and Loading. Specifically, the data in the business system is extracted, and different data are transformed and analyzed according to business needs to obtain the target data, and then the target data is loaded into the data warehouse.

[0003] In the scenario where the data warehouse is applied to the brand advertising delivery platform, the brand advertising delivery platform usually requires an offline data warehouse for brand advertising and a real-time data warehouse for brand advertising. The offline data warehouse for brand advertising is convenient for brand advertisers to view daily and hourly reports of brand advertising on the brand advertising delivery platform. The real-time data warehouse for brand advertising is convenient for brand advertisers to view minute-level and second-level reports of brand advertising on the brand advertising delivery platform. However, the offline data warehouse for brand advertising and the real-time data warehouse for brand advertising each use a set of ETL parsing logic. In this way, two sets of ETL parsing logic need to be developed and operated and maintained, which not only increases the operation and maintenance cost of the ETL parsing logic, but also increases the development cycle of the ETL parsing logic, thereby greatly increasing the labor cost.

[0004] It can be seen that how to reduce the operation and maintenance cost of ETL parsing logic and shorten the development cycle of ETL parsing logic has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of this application is to propose a data processing method, device, electronic device and computer-readable medium to solve the technical problems existing in the prior art of how to reduce the operation and maintenance cost of ETL parsing logic and shorten the development cycle of ETL parsing logic.

[0006] According to a first aspect of an embodiment of the present application, a data processing method is provided. The method comprises: uniformly parsing business object data acquired in real time from different data sources to obtain business object structured data for generating a business object data table; loading the business object structured data into a first data warehouse and a second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data.

[0007] According to a second aspect of an embodiment of the present application, a data processing method is provided. The method includes: a first data warehouse acquires business object structured data loaded by an ETL tool for generating a first business object data table, wherein the business object structured data is obtained by the ETL tool through unified parsing of business object data acquired in real time from different data sources; the first data warehouse generates the first business object data table in real time according to the business object structured data; and the first data warehouse imports the first business object data table into a business object delivery platform for display.

[0008] According to a third aspect of an embodiment of the present application, a data processing method is provided. The method includes: a second data warehouse acquires business object structured data loaded by an ETL tool for generating a second business object data table within a preset time period, wherein the business object structured data is obtained by the ETL tool through unified parsing of business object data acquired in real time from different data sources; the second data warehouse generates the second business object data table offline according to the business object structured data; and the second data warehouse imports the second business object data table into a business object delivery platform for display.

[0009] According to a fourth aspect of an embodiment of the present application, a data processing method is provided. The method includes: receiving a business object data table generation trigger operation input by a user, and displaying information of business object structured data used to generate a business object data table according to the trigger operation; and according to a selection operation of the user on the displayed business object structured data information, sending the business object structured data selected by the selection operation to a first data warehouse and a second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table in a preset time period offline according to the business object structured data.

[0010] According to a fifth aspect of an embodiment of the present application, a data processing device is provided. The device includes: a parsing module, which is used to uniformly parse business object data acquired in real time from different data sources to obtain business object structured data for generating a business object data table; a loading module, which is used to load the business object structured data into a first data warehouse and a second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data.

[0011] According to a sixth aspect of an embodiment of the present application, a data processing device is provided. The device includes: a first acquisition module, used to acquire business object structured data loaded by an ETL tool for generating a first business object data table, wherein the business object structured data is obtained by the ETL tool performing a unified analysis on business object data acquired in real time from different data sources; a first generation module, used to generate the first business object data table in real time according to the business object structured data; and a first import module, used to import the first business object data table into a business object delivery platform for display.

[0012] According to a seventh aspect of an embodiment of the present application, a data processing device is provided. The device includes: a second acquisition module, which is used to acquire business object structured data loaded by an ETL tool for generating a second business object data table within a preset time period, wherein the business object structured data is obtained by the ETL tool through unified parsing of business object data acquired in real time from different data sources; a third generation module, which is used to generate the second business object data table offline according to the business object structured data; and a second import module, which is used to import the second business object data table into a business object delivery platform for display.

[0013] According to the eighth aspect of the embodiment of the present application, a data processing device is provided. The device includes: a first receiving module, which is used to receive a business object data table generation trigger operation input by a user, and display information of business object structured data used to generate a business object data table according to the trigger operation; a selection module, which is used to send the business object structured data selected by the selection operation to a first data warehouse and a second data warehouse respectively according to the user's selection operation on the displayed business object structured data information, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table in a preset time period according to the business object structured data offline.

[0014] According to the ninth aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors; a computer-readable medium configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in the first aspect, the second aspect, the third aspect or the fourth aspect of the above-mentioned embodiments.

[0015] According to the tenth aspect of the embodiments of the present application, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method described in the first aspect, the second aspect, the third aspect or the fourth aspect of the above-mentioned embodiments is implemented.

[0016] Through the data processing solution provided in the embodiment of the present application, the business object data obtained in real time from different data sources are uniformly parsed to obtain business object structured data for generating business object data tables, and the business object structured data are loaded into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates business object data tables in real time according to the business object structured data, and the second data warehouse generates business object data tables within a preset time period offline according to the business object structured data. Compared with other existing methods, by uniformly parsing the business object data obtained in real time from different data sources, only one set of ETL parsing logic needs to be developed, operated and maintained, without the need to develop, operate and maintain multiple sets of ETL parsing logic, which not only effectively reduces the operation and maintenance cost of the ETL parsing logic, but also effectively shortens the development cycle of the ETL parsing logic, greatly reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1A This is a flowchart of the steps of the data processing method in Embodiment 1 of the present application;

[0019] Figure 1B A schematic diagram of a data processing process provided according to Embodiment 1 of the present application;

[0020] Figure 2A This is a flowchart of the steps of the data processing method in Embodiment 2 of the present application;

[0021] Figure 2B A schematic diagram of a data processing process provided according to Embodiment 2 of the present application;

[0022] Figure 3 This is a flowchart of the steps of the data processing method in Embodiment 3 of the present application;

[0023] Figure 4A This is a flowchart of the steps of the data processing method in Embodiment 3 of the present application;

[0024] Figure 4B A schematic diagram of a data processing process provided according to Embodiment 3 of the present application;

[0025] Figure 5 This is a schematic diagram of the structure of the data processing device in Embodiment 5 of the present application;

[0026] Figure 6 This is a schematic diagram of the structure of the data processing device in Embodiment 6 of the present application;

[0027] Figure 7This is a structural diagram of a data processing device in Embodiment 7 of the present application;

[0028] Figure 8 This is a schematic diagram of the structure of a data processing device in Embodiment 8 of the present application;

[0029] Fig. 9 This is a schematic diagram of the structure of an electronic device in Embodiment 9 of the present application;

[0030] Fig.10 This is the hardware structure of the electronic device in the tenth embodiment of the present application. DETAILED DESCRIPTION

[0031] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only configured to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] Reference Figure 1A , showing a step flow chart of the data processing method in Example 1 of the present application.

[0034] This embodiment describes the data processing method provided by this embodiment from the perspective of ETL tools. Specifically, the data processing method provided by this embodiment includes the following steps:

[0035] In step S101, the business object data acquired in real time from different data sources are uniformly parsed to obtain business object structured data for generating a business object data table.

[0036] In this embodiment, the data source may be a message queue for storing business object data, a database for storing business object data, or a message stack for storing business object data, etc., and this embodiment does not impose any limitation on this. The business object data includes at least one of the following: operation data for the business object, regional data for operating the business object, and delivery data for the business object. The operation data for the business object includes at least one of the following: display operation data for the business object, click operation data for the business object, purchase operation data for the commodity associated with the business object, request operation data for the business object, and delivery operation data for the business object. The regional data for operating the business object may be North China, Central China, South China, East China, etc. The delivery data for the business object may include the delivery time of the business object, the delivery time period of the business object, the delivery fee of the business object, the identifier of the business object deliverer, etc. Among them, the display operation data for the business object may include the display time for the business object, the display time period for the business object, the display times for the business object, etc. The click operation data for the business object may include the number of clicks on the business object, the time point of clicks on the business object, the user data for clicking on the business object, etc. The purchase operation data for the goods associated with the business object may include the data of the purchased goods, the time point of purchasing the goods, the cost of purchasing the goods, the regional data for purchasing the goods, etc. The request operation data for the business object may include the number of requests for the data associated with the business object, the time point of request for the data associated with the business object, the user data for requesting the data associated with the business object, etc. The delivery operation data for the business object may include the number of deliveries for the data associated with the business object, the time point of delivery for the data associated with the business object, etc. Among them, the business object may be a brand advertisement placed by a brand advertiser on a brand advertisement delivery platform. The business object data may be the data of a brand advertisement placed on a brand advertisement delivery platform. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0037] In this embodiment, the business object data table includes a business object delivery data table, a business object quantitative delivery data table, a business object delivery financial report, etc. The business object structured data can be understood as the structured data of the business object data, for example, type data for business object data identification, etc. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0038] In some optional embodiments, before the unified parsing of the business object data obtained in real time from different data sources, the method further includes: cleaning the business object data obtained in real time from different data sources to obtain the cleaned business object data; the unified parsing of the business object data obtained in real time from different data sources includes: unified parsing of the cleaned business object data to obtain the business object structured data used to generate the business object data table. Thus, by cleaning the business object data obtained in real time from different data sources, unnecessary and non-compliant data in the business object data can be cleaned out. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0039] In a specific example, when cleaning the business object data obtained in real time from different data sources, the null value is replaced with a specific value or directly filtered out according to the needs of the business object delivery business. The business object data can also be cleaned by verifying the correctness of the data. Specifically, the data that does not conform to the meaning of the business object delivery is processed, for example, the string in a field representing the quantity is replaced with 0, the non-date string of a date field is filtered out, and so on. In addition, the business object data can also be cleaned by data transcoding. Specifically, a field represented by a code in a source data is converted into a value representing its true meaning through an associated coding table, and so on. In addition, the business object data can also be cleaned by a unified data standard. For example, there are many ways to represent men and women in the source data. When extracting, the conversion is directly performed according to the value defined in the model to uniformly represent men and women. Of course, this embodiment is not limited to this, and data cleaning can also be defined by other business rules. It can be understood that the above description is only exemplary, and the embodiment of the present application does not make any limitation to this.

[0040] In some optional embodiments, when the business object data acquired in real time from different data sources are uniformly parsed, the data formats of the business object data acquired in real time from the different data sources are converted to obtain the business object data having a uniform data format; the business object data having a uniform data format are structured to obtain the business object structured data. Thus, by converting the data formats of the business object data acquired in real time from different data sources, the business object data having a uniform data format can be obtained, which is beneficial to the structuring of the business object data. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0041] In a specific example, the data formats of the business object data obtained in real time from different data sources are various and not uniform. Therefore, it is necessary to convert the data formats of the business object data obtained in real time from different data sources to obtain business object data with a uniform data format. For example, all dates are formatted into the format of yyyy-MM-dd HH:mm:ss. It is understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0042] In some optional embodiments, the business object structured data includes type data for identifying the business object data. When the business object data having a uniform data format is structured, the business object data having a uniform data format is type-identified to obtain type data for identifying the business object data. Thus, by type-identifying the business object data having a uniform data format, type data for identifying the business object data can be obtained, thereby facilitating the generation of a business object data table. It is to be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0043] In a specific example, when the business object data includes display operation data for the business object, the type data identified for the business object data may be a display type. When the business object data includes click operation data for the business object, the type data identified for the business object data may be a click type. When the business object data includes purchase operation data for commodities associated with the business object, the type data identified for the business object data may be a consumption type. When the business object data includes request operation data for the business object, the type data identified for the business object data may be a request type. When the business object data includes issuing operation data for the business object, the type data identified for the business object data may be an issuing type. It will be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0044] In some optional embodiments, when the business object data having a unified data format is type-identified, classification configuration parameters are set according to the preset classification rules of the business object delivery service; a classification integration template is generated according to the classification configuration parameters; an SQL (Structured Query Language) code is generated according to the classification integration template and the multi-source data integration framework; and the business object data having a unified data format is type-identified through the SQL code. Thus, the business object data having a unified data format can be conveniently type-identified through the generated SQL code. It is to be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0045] In step S102, the business object structured data is loaded into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data.

[0046] In an embodiment of the present application, the first data warehouse is used to generate a business object data table in real time according to the business object structured data, and the function of the first data warehouse can be implemented by the Blink tool. The second data warehouse is used to generate a business object data table within a preset time period offline according to the business object structured data, and the function of the second data warehouse can be implemented by an open data processing service (Open Data Processing Service, ODPS). Among them, the preset time period can be set by a technician in this field according to actual needs, for example, one day or one hour, and this embodiment does not impose any restrictions on this. The business object data table includes a business object delivery data table, a business object quantitative delivery data table, a business object delivery financial report, etc. The business object delivery data table contains content data such as the revenue data of the business object delivered to the platform, the display time of the business object delivered to the platform, and the number of clicks on the business object delivered to the platform. The business object quantitative delivery data table contains content data such as the revenue data of the business object delivered to the platform within the scope of the business object delivery fee, the display time of the business object delivered to the platform, and the number of clicks on the business object delivered to the platform. The business object delivery financial report contains content data such as the amount of revenue delivered to the platform by the business object, the revenue batch delivered to the platform by the business object, etc. It should be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation thereto.

[0047] In a specific example, Figure 1BAs shown, the ETL tool obtains business object data from data source 1, data source 2 and data source 3 in real time, and uniformly parses the business object data obtained from data source 1, data source 2 and data source 3 in real time to obtain business object structured data for generating business object data tables, and then loads the business object structured data into the first data warehouse and the second data warehouse respectively. The first data warehouse generates the business object data table in real time according to the business object structured data, and imports the real-time generated business object data table into the business object delivery platform for display. The second data warehouse generates the business object data table within a preset time period offline according to the business object structured data, and imports the business object data table within the preset time period generated offline into the business object delivery platform for display. Among them, the ETL tool can be understood as a tool that executes ETL logic. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0048] Through the data processing method provided in the embodiment of the present application, the business object data obtained in real time from different data sources are uniformly parsed to obtain business object structured data for generating business object data tables, and the business object structured data are loaded into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates business object data tables in real time according to the business object structured data, and the second data warehouse generates business object data tables within a preset time period offline according to the business object structured data. Compared with other existing methods, by uniformly parsing the business object data obtained in real time from different data sources, only one set of ETL parsing logic needs to be developed, operated and maintained, without the need to develop, operate and maintain multiple sets of ETL parsing logic, which not only effectively reduces the operation and maintenance cost of the ETL parsing logic, but also effectively shortens the development cycle of the ETL parsing logic, greatly reducing the labor cost.

[0049] The data processing method of this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices, etc.

[0050] Reference Figure 2A , showing a step flow chart of the data processing method in Example 2 of the present application.

[0051] This embodiment describes the data processing method provided by this embodiment from the perspective of the first data warehouse. Specifically, the data processing method provided by this embodiment includes the following steps:

[0052] In step S201, the first data warehouse obtains business object structured data loaded by an ETL tool for generating a first business object data table.

[0053] In this embodiment, the first data warehouse is used to generate a business object data table in real time based on the business object structured data, and the function of the first data warehouse can be implemented by the Blink tool. The ETL tool can be understood as a tool for executing ETL logic. The business object structured data is obtained by the ETL tool through unified parsing of the business object data obtained in real time from different data sources. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitations on this.

[0054] In step S202, the first data warehouse generates the first business object data table in real time according to the business object structured data.

[0055] In some optional embodiments, when the first data warehouse generates the first business object data table in real time based on the business object structured data, the business object structured data is stored through the storage layer of the first data warehouse; the business object structured data is counted through the statistical layer of the first data warehouse to obtain detailed data of the business object structured data; the detailed data of the business object structured data is summarized through the summary layer of the first data warehouse to obtain the summary result of the business object structured data; the summary result of the business object structured data is applied through the application layer of the first data warehouse to generate the first business object data table in real time. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0056] In a specific example, in addition to summarizing the detailed data of the business object structured data, the summary layer of the first data warehouse can also perform public indicator processing on the detailed data of the business object structured data. In addition to applying the summary results of the business object structured data, the application layer of the first data warehouse can also perform personalized indicator processing on the summary results of the business object structured data. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0057] In step S203, the first data warehouse imports the first business object data table into the business object delivery platform for display.

[0058] In this embodiment, the business object delivery platform can be understood as a platform for delivering business objects. It can be understood that the above description is only exemplary, and the embodiment of the present application does not impose any limitation on this.

[0059] In some optional embodiments, the method further includes: when the first data warehouse fails to generate the first business object data table in real time, the first data warehouse obtains the second business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period, and generates the first business object data table in real time according to the second business object data table. Thereby, when the first data warehouse fails to generate the first business object data table in real time, the second business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period can be used to generate the first business object data table in real time, thereby greatly improving the stability of the data. It will be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0060] In a specific example, the second business object data table within a preset time period generated offline by the second data warehouse based on the business object structured data includes a minute-level business object data table and a second-level business object data table. Therefore, when a failure occurs in the real-time generation of the first business object data table by the first data warehouse, the minute-level business object data table and the second-level business object data table included in the second business object data table can be used to generate the first business object data table in real time, thereby greatly improving the stability of the data. It will be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0061] In a specific example, Figure 2BAs shown, the ETL tool obtains the dotting data from the message queue (MQ). The dotting data can be understood as data used to record operations on business objects. After obtaining the dotting data, the ETL tool uniformly parses the dotting data to obtain business object structured data for generating a business object data table, and temporarily stores the obtained business object structured data in the storage middleware. Then, through the storage middleware, the business object structured data is loaded into the first data warehouse (implemented by the Blink tool) and the second data warehouse (implemented by the ODPS tool) respectively. Among them, the business object structured data includes business object operation data with type identification, and the type identification includes consumption type, request type, issuance type, display type, click type, etc. After the business object structured data is loaded into the first data warehouse respectively, the business object operation data with type identification is stored through the ODS (operational data store) layer of the first data warehouse. For example, business object operation data with consumption type, business object operation data with request type, business object operation data with dispatch type, business object operation data with display type, business object operation data with click type, etc. Then, through the DWD (Data Warehouse Detail) layer of the first data warehouse, the business object operation data with type identification is counted to obtain detailed data of the business object operation data with type identification. For example, detailed data of business object operation data with consumption type, detailed data of business object operation data with request type, detailed data of business object operation data with dispatch type, detailed data of business object operation data with display type, detailed data of business object operation data with click type, etc. Then, through the DWS (Data Warehouse Summary) layer of the first data warehouse, the detailed data of the business object operation data with type identification is summarized to obtain the summary result of the business object operation data with type identification. For example, the summary results of the business object operation data with consumption type, the summary results of the business object operation data with request type, the summary results of the business object operation data with distribution type, the summary results of the business object operation data with display type, the summary results of the business object operation data with click type, etc. Finally, through the ADS (Application Data store) layer of the first data warehouse, the summary results of the business object operation data with type identification are applied to generate the first business object data table in real time. For example, the business object delivery data table, the business object quantitative delivery data table, and the business object financial report.After generating the first business object data table in real time, the first data warehouse imports the first business object data table generated in real time into the business object delivery platform for display. After loading the business object structured data into the second data warehouse respectively, the second data warehouse generates a second business object data table within a preset time period offline based on the business object structured data in the same manner as the first data warehouse. After generating the second business object data table within a preset time period offline, the second data warehouse imports the second business object data table generated offline into the business object delivery platform for display. It will be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0062] Through the data processing method provided in the embodiment of the present application, the first data warehouse obtains the business object structured data loaded by the ETL tool for generating the first business object data table, the business object structured data is obtained by the ETL tool performing unified parsing on the business object data obtained in real time from different data sources, and the first business object data table is generated in real time according to the business object structured data, and then the first business object data table is imported into the business object delivery platform for display. Compared with other existing methods, by performing unified parsing on the business object data obtained in real time from different data sources, only one set of ETL parsing logic needs to be developed, operated and maintained, instead of multiple sets of ETL parsing logic needs to be developed, operated and maintained, which not only effectively reduces the operation and maintenance cost of the ETL parsing logic, but also effectively shortens the development cycle of the ETL parsing logic, greatly reducing the labor cost.

[0063] The data processing method of this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices, etc.

[0064] Reference Figure 3 , showing a step flow chart of the data processing method in Example 3 of the present application.

[0065] This embodiment describes the data processing method provided by this embodiment from the perspective of the second data warehouse. Specifically, the data processing method provided by this embodiment includes the following steps:

[0066] In step S301, the second data warehouse obtains business object structured data loaded by an ETL tool for generating a second business object data table within a preset time period.

[0067] In this embodiment, the second data warehouse is used to generate a business object data table within a preset time period offline based on the business object structured data, and the function of the second data warehouse can be implemented through the open data processing service (OpenData Processing Service, ODPS). The ETL tool can be understood as a tool for executing ETL logic. The business object structured data is obtained by the ETL tool performing a unified analysis of the business object data acquired in real time from different data sources. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitations on this.

[0068] In step S302, the second data warehouse generates the second business object data table offline according to the business object structured data.

[0069] In some optional embodiments, when the second data warehouse generates the second business object data table offline based on the business object structured data, the business object structured data is stored through the storage layer of the second data warehouse; the business object structured data is counted through the statistical layer of the second data warehouse to obtain detailed data of the business object structured data; the detailed data of the business object structured data is summarized through the summary layer of the second data warehouse to obtain the summary result of the business object structured data; the summary result of the business object structured data is applied through the application layer of the second data warehouse to generate the second business object data table offline. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0070] In a specific example, in addition to summarizing the detailed data of the business object structured data, the summary layer of the second data warehouse can also perform public indicator processing on the detailed data of the business object structured data. In addition to applying the summary results of the business object structured data, the application layer of the second data warehouse can also perform personalized indicator processing on the summary results of the business object structured data. It can be understood that the above description is only exemplary and the embodiments of the present application do not impose any limitation on this.

[0071] In step S303, the second data warehouse imports the second business object data table into the business object delivery platform for display.

[0072] In this embodiment, the business object delivery platform can be understood as a platform for delivering business objects. It can be understood that the above description is only exemplary, and the embodiment of the present application does not impose any limitation on this.

[0073] In some optional embodiments, the method further includes: when the second data warehouse fails to generate the second business object data table offline, the second data warehouse obtains the first business object data table generated in real time by the first data warehouse based on the business object structured data, and generates the second business object data table offline based on the first business object data table. Thereby, when the second data warehouse fails to generate the second business object data table within a preset time period offline, the first business object data table generated in real time by the first data warehouse based on the business object structured data can be used to generate the second business object data table within a preset time period offline, thereby greatly improving the stability of the data. It will be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0074] In a specific example, the first business object data table generated in real time by the first data warehouse based on the business object structured data is a minute-level business object data table and a second-level business object data table. Therefore, when the second data warehouse fails to generate the second business object data table within the preset time period offline, the first business object data table can be used to generate the second business object data table within the preset time period offline, thereby greatly improving the stability of the data. Among them, the preset time period can be set by those skilled in the art according to actual needs, and the embodiments of the present application do not impose any limitations on this. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitations on this.

[0075] Through the data processing method provided in the embodiment of the present application, the second data warehouse obtains the business object structured data loaded by the ETL tool for generating the second business object data table within a preset time period, the business object structured data is obtained by the ETL tool through unified parsing of the business object data obtained in real time from different data sources, and the second business object data table is generated offline according to the business object structured data, and then the second business object data table is imported into the business object delivery platform for display. Compared with other existing methods, by uniformly parsing the business object data obtained in real time from different data sources, only one set of ETL parsing logic needs to be developed, operated and maintained, without the need to develop, operate and maintain multiple sets of ETL parsing logic, which not only effectively reduces the operation and maintenance cost of the ETL parsing logic, but also effectively shortens the development cycle of the ETL parsing logic, greatly reducing the labor cost.

[0076] The data processing method of this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices, etc.

[0077] Reference Figure 4A , showing a step flow chart of the data processing method in Example 4 of the present application.

[0078] This embodiment describes the data processing method provided by this embodiment from the perspective of the visual interface of the ETL tool. Specifically, the data processing method provided by this embodiment includes the following steps:

[0079] In step S401, a business object data table generation trigger operation input by a user is received, and information of business object structured data used to generate a business object data table is displayed according to the trigger operation.

[0080] In an embodiment of the present application, the business object data table generation trigger operation input by the user may be a click operation of the user on the business object data table generation control of the visual interface of the ETL tool. Wherein, the business object data table generation control is used to indicate the generation of a business object data table when operated. After the user inputs the business object data table generation trigger operation in the visual interface of the ETL tool, the ETL tool uniformly parses the business object data obtained in real time from different data sources to obtain business object structured data for generating the business object data table, and displays the information of the business object structured data for generating the business object data table in the visual interface of the ETL tool. Wherein, the information of the business object structured data may be a type identifier of the business object operation data, and the type identifier includes consumption type, request type, issuance type, display type, click type, etc. It can be understood that the above description is only exemplary, and the embodiment of the present application does not impose any limitation on this.

[0081] In some optional embodiments, before sending the business object structured data selected by the selection operation to the first data warehouse and the second data warehouse respectively, the method further includes: receiving the information of the first data warehouse and the second data warehouse input by the user, and determining the first data warehouse and the second data warehouse to which the selected business object structured data is to be sent respectively based on the information of the first data warehouse and the information of the second data warehouse. Among them, the information of the first data warehouse may be the name or identifier of the first data warehouse, and the information of the second data warehouse may be the name or identifier of the second data warehouse. Thereby, the first data warehouse and the second data warehouse to which the selected business object structured data is to be sent can be accurately determined through the information of the first data warehouse and the information of the second data warehouse input by the user. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0082] In step S402, based on the user's selection operation on the displayed business object structured data, the business object structured data selected by the selection operation is sent to the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time based on the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline based on the business object structured data.

[0083] In an embodiment of the present application, the first data warehouse is used to generate a business object data table in real time according to the business object structured data, and the function of the first data warehouse can be implemented by the Blink tool. The second data warehouse is used to generate a business object data table within a preset time period offline according to the business object structured data, and the function of the second data warehouse can be implemented by an open data processing service (Open Data Processing Service, ODPS). Among them, the preset time period can be set by a technician in this field according to actual needs, for example, one day or one hour, and this embodiment does not impose any restrictions on this. The business object data table includes a business object delivery data table, a business object quantitative delivery data table, a business object delivery financial report, etc. The business object delivery data table contains content data such as the revenue data of the business object delivered to the platform, the display time of the business object delivered to the platform, and the number of clicks on the business object delivered to the platform. The business object quantitative delivery data table contains content data such as the revenue data of the business object delivered to the platform within the scope of the business object delivery fee, the display time of the business object delivered to the platform, and the number of clicks on the business object delivered to the platform. The business object delivery financial report contains content data such as the amount of revenue delivered to the platform by the business object, the revenue batch delivered to the platform by the business object, etc. It should be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation thereto.

[0084] In some optional embodiments, the method further includes: if failure information of the first data warehouse generating the business object data table in real time according to the business object structured data is received, then obtaining and displaying the information of the business object data table generated offline according to the business object structured data within a preset time period by the second data warehouse, and sending the business object data table indicated by the information of the business object data table to the first data warehouse, so that the first data warehouse generates the business object data table in real time. Among them, the information of the business object data table may be the name or identifier of the business object data table. Thereby, when the first data warehouse fails to generate the first business object data table in real time, the second business object data table generated offline according to the business object structured data within a preset time period by the second data warehouse can be used to generate the first business object data table in real time, thereby greatly improving the stability of the data. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0085] In a specific example, the second business object data table within a preset time period generated offline by the second data warehouse based on the business object structured data includes a minute-level business object data table and a second-level business object data table. Therefore, when a failure occurs in the real-time generation of the first business object data table by the first data warehouse, the minute-level business object data table and the second-level business object data table included in the second business object data table can be used to generate the first business object data table in real time, thereby greatly improving the stability of the data. It will be understood that the above description is merely exemplary and the embodiments of the present application do not impose any limitation on this.

[0086] In some optional embodiments, the method further includes: if failure information is received that the second data warehouse generates the business object data table offline according to the business object structured data, then obtaining and displaying the information of the business object data table generated in real time by the first data warehouse according to the business object structured data, and sending the business object data table indicated by the information of the business object data table to the second data warehouse, so that the second data warehouse generates the business object data table offline. Wherein, the information of the business object data table may be the name or identifier of the business object data table. Thereby, when the second data warehouse fails to generate the second business object data table within a preset time period offline, the first business object data table generated in real time by the first data warehouse according to the business object structured data can be used to generate the second business object data table within a preset time period offline, thereby greatly improving the stability of the data. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitation on this.

[0087] In a specific example, the first business object data table generated in real time by the first data warehouse based on the business object structured data is a minute-level business object data table and a second-level business object data table. Therefore, when the second data warehouse fails to generate the second business object data table within the preset time period offline, the first business object data table can be used to generate the second business object data table within the preset time period offline, thereby greatly improving the stability of the data. Among them, the preset time period can be set by those skilled in the art according to actual needs, and the embodiments of the present application do not impose any limitations on this. It can be understood that the above description is only exemplary, and the embodiments of the present application do not impose any limitations on this.

[0088] In a specific example, Figure 4B As shown, after the user inputs the trigger operation of generating a business object data table in the visual interface of the ETL tool, the ETL tool uniformly parses the business object data obtained in real time from different data sources to obtain the business object structured data used to generate the business object data table, and displays the information of the business object structured data used to generate the business object data table in the visual interface of the ETL tool, such as consumption type, request type, issuance type, display type, click type, etc. After the user selects the click type on the visual interface of the ETL tool, according to the user's selection operation on the displayed click mark, the business object operation data with the click mark selected by the selection operation is sent to the first data warehouse and the second data warehouse respectively. The first data warehouse generates a business object data table in real time according to the business object structured data, and imports the business object data table generated in real time into the business object delivery platform for display. The second data warehouse generates a business object data table within a preset time period offline according to the business object structured data, and imports the business object data table generated offline within the preset time period into the business object delivery platform for display. Among them, the ETL tool can be understood as a tool for executing ETL logic. It can be understood that the above description is only exemplary, and the embodiments of the present application do not make any limitation to this.

[0089] Through the data processing method provided in the embodiment of the present application, a business object data table generation trigger operation input by a user is received, information of business object structured data used to generate the business object data table is displayed according to the trigger operation, and then according to the user's selection operation on the displayed business object structured data information, the business object structured data selected by the selection operation is sent to the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates a business object data table in real time according to the business object structured data, and the second data warehouse generates a business object data table within a preset time period offline according to the business object structured data. Compared with other existing methods, it is convenient for users to interact with the visual interface of the ETL tool.

[0090] The data processing method of this embodiment can be executed by any appropriate device with data processing capabilities, including but not limited to: cameras, terminals, mobile terminals, PCs, servers, vehicle-mounted devices, entertainment equipment, advertising equipment, personal digital assistants (PDAs), tablet computers, laptop computers, handheld game consoles, smart glasses, smart watches, wearable devices, virtual display devices or display enhancement devices, etc.

[0091] Reference Figure 5 , showing a schematic diagram of the structure of the data processing device in Example 5 of the present application.

[0092] The data processing device provided in this embodiment includes: a parsing module 502, which is used to uniformly parse the business object data obtained in real time from different data sources to obtain business object structured data for generating a business object data table; a loading module 503, which is used to load the business object structured data to the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data.

[0093] Optionally, before the parsing module 502, the device also includes: a cleaning module 501, which is used to clean the business object data obtained in real time from different data sources to obtain the cleaned business object data; the parsing module 502 includes: a parsing sub-module 5021, which is used to uniformly parse the cleaned business object data to obtain the business object structured data used to generate the business object data table.

[0094] Optionally, the parsing module 502 includes: a conversion submodule 5022, used to convert the data format of the business object data obtained in real time from different data sources to obtain the business object data with a unified data format; a structuring submodule 5023, used to structure the business object data with a unified data format to obtain the business object structured data.

[0095] Optionally, the business object structured data includes type data for identifying the business object data, and the structuring submodule 5023 is specifically used to: perform type identification on the business object data having a unified data format to obtain type data for identifying the business object data.

[0096] Optionally, the business object data includes at least one of the following: operation data for the business object, regional data for operating the business object, and delivery data for the business object.

[0097] Optionally, the operation data for the business object includes at least one of the following: display operation data for the business object, click operation data for the business object, purchase operation data for goods associated with the business object, request operation data for the business object, and delivery operation data for the business object.

[0098] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0099] Reference Figure 6 , showing a schematic diagram of the structure of the data processing device in Example 6 of the present application.

[0100] The data processing device provided in this embodiment includes: a first acquisition module 601, used to acquire business object structured data loaded by an ETL tool for generating a first business object data table, wherein the business object structured data is obtained by the ETL tool performing a unified analysis on business object data acquired in real time from different data sources; a first generation module 602, used to generate the first business object data table in real time according to the business object structured data; and a first import module 603, used to import the first business object data table into a business object delivery platform for display.

[0101] Optionally, the first generation module 602 is specifically used to: store the business object structured data through the storage layer of the first data warehouse; perform statistics on the business object structured data through the statistical layer of the first data warehouse to obtain detailed data of the business object structured data; summarize the detailed data of the business object structured data through the summary layer of the first data warehouse to obtain a summary result of the business object structured data; apply the summary result of the business object structured data through the application layer of the first data warehouse to generate the first business object data table in real time.

[0102] Optionally, the device also includes: a second generation module 604, which is used to obtain a second business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period when a failure occurs in the first data warehouse generating the first business object data table in real time, and generate the first business object data table in real time based on the second business object data table.

[0103] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0104] Reference Figure 7 , showing a schematic diagram of the structure of the data processing device in Example 7 of the present application.

[0105] The data processing device provided in this embodiment includes: a second acquisition module 701, used to acquire business object structured data loaded by the ETL tool for generating a second business object data table within a preset time period, wherein the business object structured data is obtained by the ETL tool through unified parsing of business object data acquired in real time from different data sources; a third generation module 702, used to generate the second business object data table offline according to the business object structured data; and a second import module 703, used to import the second business object data table into a business object delivery platform for display.

[0106] Optionally, the third generation module 702 is specifically used to: store the business object structured data through the storage layer of the second data warehouse; perform statistics on the business object structured data through the statistical layer of the second data warehouse to obtain detailed data of the business object structured data; summarize the detailed data of the business object structured data through the summary layer of the second data warehouse to obtain a summary result of the business object structured data; and apply the summary result of the business object structured data through the application layer of the second data warehouse to generate the second business object data table.

[0107] Optionally, the device also includes: a fourth generation module 704, which is used to obtain the first business object data table generated in real time by the first data warehouse according to the business object structured data when the second data warehouse fails to generate the second business object data table offline, and generate the second business object data table offline according to the first business object data table.

[0108] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0109] Reference Figure 8 , showing a schematic diagram of the structure of the data processing device in Example 8 of the present application.

[0110] The data processing device provided in this embodiment includes: a first receiving module 801, which is used to receive a business object data table generation trigger operation input by a user, and display information of business object structured data used to generate a business object data table according to the trigger operation; a selection module 803, which is used to send the business object structured data selected by the selection operation to the first data warehouse and the second data warehouse respectively according to the user's selection operation on the displayed business object structured data information, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and enables the second data warehouse to generate the business object data table within a preset time period offline according to the business object structured data.

[0111] Optionally, before the selection module 803, the device also includes: a second receiving module 802, which is used to receive the information of the first data warehouse and the information of the second data warehouse input by the user, and determine the first data warehouse and the second data warehouse to which the selected business object structured data is to be sent based on the information of the first data warehouse and the information of the second data warehouse.

[0112] Optionally, the device also includes: a first execution module 804, which is used to obtain and display information of the business object data table generated by the second data warehouse offline according to the business object structured data within a preset time period if fault information is received that the first data warehouse generates the business object data table in real time according to the business object structured data, and send the business object data table indicated by the information of the business object data table to the first data warehouse, so that the first data warehouse generates the business object data table in real time; or, a second execution module 805, which is used to obtain and display information of the business object data table generated by the first data warehouse in real time according to the business object structured data if fault information is received that the second data warehouse generates the business object data table offline according to the business object structured data, and send the business object data table indicated by the information of the business object data table to the second data warehouse, so that the second data warehouse generates the business object data table offline.

[0113] The data processing device of this embodiment is used to implement the corresponding data processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0114] Fig. 9 This is a schematic diagram of the structure of an electronic device in Embodiment 9 of the present application; the electronic device may include:

[0115] One or more processors 901;

[0116] The computer readable medium 902 may be configured to store one or more programs,

[0117] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in the above-mentioned embodiment 1, embodiment 2, embodiment 3 or embodiment 4.

[0118] Fig.10 The hardware structure of the electronic device in the tenth embodiment of the present application; Fig.10 As shown, the hardware structure of the electronic device may include: a processor 1001, a communication interface 1002, a computer readable medium 1003 and a communication bus 1004;

[0119] The processor 1001, the communication interface 1002, and the computer-readable medium 1003 communicate with each other via a communication bus 1004;

[0120] Optionally, the communication interface 1002 may be an interface of a communication module, such as an interface of a GSM module;

[0121] Among them, the processor 1001 can be specifically configured to: uniformly parse the business object data obtained in real time from different data sources to obtain business object structured data for generating a business object data table; load the business object structured data into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data. In addition, the processor 1001 can also be configured to: obtain the business object structured data loaded by the ETL tool for generating the first business object data table, the business object structured data is obtained by the ETL tool uniformly parsing the business object data obtained in real time from different data sources; generate the first business object data table in real time according to the business object structured data; import the first business object data table into the business object delivery platform for display. In addition, the processor 1001 can also be configured to: obtain business object structured data loaded by the ETL tool for generating a second business object data table within a preset time period, the business object structured data is obtained by the ETL tool performing a unified analysis on the business object data obtained in real time from different data sources; offline generate the second business object data table according to the business object structured data; import the second business object data table into the business object delivery platform for display. In addition, the processor 1001 can also be configured to: receive a business object data table generation trigger operation input by a user, display information of the business object structured data used to generate the business object data table according to the trigger operation; according to the user's selection operation on the displayed business object structured data information, send the business object structured data selected by the selection operation to the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data.

[0122] The processor 1001 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0123] The computer readable medium 1003 can be, but is not limited to, a random access memory medium (RAM), a read-only memory medium (ROM), a programmable read-only memory medium (PROM), an erasable read-only memory medium (EPROM), an electrically erasable read-only memory medium (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0124] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code configured to execute the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program configured to be used by or in combination with an instruction execution system, an apparatus or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0125] Computer program code configured to perform the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions configured to implement the specified logical function. There are specific sequential relationships in the above-mentioned specific embodiments, but these sequential relationships are only exemplary. When the specific implementation is implemented, these steps may be less, more, or the execution order may be adjusted. That is, in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The modules involved in the embodiments of the present application may be implemented by software or hardware. The modules described may also be set in a processor, for example, may be described as: a processor includes a parsing module and a loading module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the parsing module may also be described as "a module for uniformly parsing business object data acquired in real time from different data sources to obtain business object structured data for generating business object data tables".

[0128] As another aspect, the present application further provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method described in the above-mentioned Embodiment 1, Embodiment 2, Embodiment 3 or Embodiment 4.

[0129] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: uniformly parses the business object data obtained in real time from different data sources to obtain business object structured data for generating a business object data table; loads the business object structured data into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data. In addition, the device is also enabled to: obtain the business object structured data loaded by the ETL tool for generating the first business object data table, the business object structured data is obtained by the ETL tool uniformly parsing the business object data obtained in real time from different data sources; generates the first business object data table in real time according to the business object structured data; and imports the first business object data table into the business object delivery platform for display. In addition, the device is also configured to: obtain business object structured data loaded by the ETL tool for generating a second business object data table within a preset time period, the business object structured data being obtained by the ETL tool through unified parsing of business object data obtained in real time from different data sources; generate the second business object data table offline according to the business object structured data; and import the second business object data table into the business object delivery platform for display. In addition, the device is also configured to: receive a business object data table generation trigger operation input by a user, and display information of the business object structured data used to generate the business object data table according to the trigger operation; and send the business object structured data selected by the selection operation to the first data warehouse and the second data warehouse respectively according to the user's selection operation on the displayed business object structured data, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data.

[0130] The expressions "first", "second", "the first" or "the second" used in various embodiments of the present disclosure may modify various components regardless of order and / or importance, but these expressions do not limit the corresponding components. The above expressions are only configured for the purpose of distinguishing an element from other elements. For example, a first user device and a second user device represent different user devices, although both are user devices. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present disclosure.

[0131] When one element (e.g., a first element) is referred to as being "(operably or communicatively) coupled" or "(operably or communicatively) coupled to" or "connected to" another element (e.g., a second element), it is understood that the one element is directly connected to the other element or the one element is indirectly connected to the other element via yet another element (e.g., a third element). Conversely, it is understood that when an element (e.g., a first element) is referred to as being "directly connected" or "directly coupled" to another element (the second element), no element (e.g., a third element) is interposed between the two.

[0132] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data processing method, the method comprising: Perform unified parsing of business object data acquired in real time from different data sources to obtain business object structured data for generating business object data tables; Loading the business object structured data into a first data warehouse and a second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data; The method further comprises: If failure information indicating that the first data warehouse generates the business object data table in real time according to the business object structured data is received, then information on the business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period is obtained and displayed, and the business object data table indicated by the information on the business object data table is sent to the first data warehouse, so that the first data warehouse generates the business object data table in real time; or, If fault information is received that the second data warehouse generates the business object data table offline based on the business object structured data, the information of the business object data table generated in real time by the first data warehouse based on the business object structured data is obtained and displayed, and the business object data table indicated by the information of the business object data table is sent to the second data warehouse, so that the second data warehouse generates the business object data table offline.

2. The method according to claim 1, wherein: Before uniformly parsing the business object data acquired in real time from different data sources, the method further includes: Cleaning the business object data acquired in real time from different data sources to obtain cleaned business object data; The unified parsing of the business object data obtained in real time from different data sources includes: The cleaned business object data is uniformly parsed to obtain the business object structured data for generating the business object data table.

3. The method according to claim 1, wherein: The unified parsing of the business object data obtained in real time from different data sources includes: Converting the data format of the business object data acquired in real time from different data sources to obtain the business object data in a unified data format; The business object data having the unified data format is structured to obtain the business object structured data.

4. The method according to claim 3, wherein: The business object structured data includes type data for identifying the business object data, The structuring of the business object data having the unified data format comprises: Type identification is performed on the business object data having the unified data format to obtain type data for identifying the business object data.

5. The method according to any one of claims 1 to 4, wherein: The business object data includes at least one of the following: operation data for the business object, regional data for operating the business object, and delivery data for the business object.

6. The method according to claim 5, wherein: The operation data for the business object includes at least one of the following: display operation data for the business object, click operation data for the business object, purchase operation data for the goods associated with the business object, request operation data for the business object, and delivery operation data for the business object.

7. A data processing method, the method comprising: The first data warehouse acquires business object structured data loaded by the ETL tool for generating a first business object data table, wherein the business object structured data is obtained by the ETL tool performing unified parsing on business object data acquired in real time from different data sources; The first data warehouse generates the first business object data table in real time according to the business object structured data; The first data warehouse imports the first business object data table into the business object delivery platform for display; The method also includes: when the first data warehouse fails to generate the first business object data table in real time, the first data warehouse obtains a second business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period, and generates the first business object data table in real time according to the second business object data table.

8. The method according to claim 7, wherein: The first data warehouse generates the first business object data table in real time according to the business object structured data, including: Storing the business object structured data through the storage layer of the first data warehouse; By using the statistical layer of the first data warehouse, statistics are collected on the business object structured data to obtain detailed data of the business object structured data; Aggregating the detailed data of the business object structured data through the aggregation layer of the first data warehouse to obtain an aggregation result of the business object structured data; The summary result of the business object structured data is applied through the application layer of the first data warehouse to generate the first business object data table in real time.

9. A data processing method, the method comprising: The second data warehouse acquires business object structured data loaded by the ETL tool for generating a second business object data table within a preset time period, wherein the business object structured data is obtained by the ETL tool through unified parsing of business object data acquired in real time from different data sources; The second data warehouse generates the second business object data table offline according to the business object structured data; The second data warehouse imports the second business object data table into the business object delivery platform for display; The method also includes: when the second data warehouse fails to generate the second business object data table offline, the second data warehouse obtains the first business object data table generated in real time by the first data warehouse according to the business object structured data, and generates the second business object data table offline according to the first business object data table.

10. The method according to claim 9, wherein: The second data warehouse generates the second business object data table offline according to the business object structured data, including: Storing the business object structured data through the storage layer of the second data warehouse; By using the statistical layer of the second data warehouse, statistics are collected on the business object structured data to obtain detailed data of the business object structured data; Aggregating the detailed data of the business object structured data through the aggregation layer of the second data warehouse to obtain an aggregation result of the business object structured data; The summary result of the business object structured data is applied through the application layer of the second data warehouse to generate the second business object data table offline.

11. A data processing method, the method comprising: Receive a business object data table generation trigger operation input by a user, and display information of business object structured data used to generate the business object data table according to the trigger operation; According to the user's selection operation on the displayed business object structured data, the business object structured data selected by the selection operation is sent to the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data; The method further comprises: If failure information indicating that the first data warehouse generates the business object data table in real time according to the business object structured data is received, then information on the business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period is obtained and displayed, and the business object data table indicated by the information on the business object data table is sent to the first data warehouse, so that the first data warehouse generates the business object data table in real time; or, If fault information is received that the second data warehouse generates the business object data table offline based on the business object structured data, the information of the business object data table generated in real time by the first data warehouse based on the business object structured data is obtained and displayed, and the business object data table indicated by the information of the business object data table is sent to the second data warehouse, so that the second data warehouse generates the business object data table offline.

12. The method according to claim 11, wherein: Before sending the business object structured data selected by the selection operation to the first data warehouse and the second data warehouse respectively, the method further includes: The information of the first data warehouse and the information of the second data warehouse input by the user are received, and the first data warehouse and the second data warehouse to which the selected business object structured data are to be sent are respectively determined according to the information of the first data warehouse and the information of the second data warehouse.

13. A data processing device, comprising: A parsing module, used to uniformly parse the business object data acquired in real time from different data sources to obtain business object structured data for generating a business object data table; A loading module, used to load the business object structured data into the first data warehouse and the second data warehouse respectively, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data; The loading module is further used for obtaining and displaying information of the business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period if failure information of the first data warehouse generating the business object data table in real time according to the business object structured data is received, and sending the business object data table indicated by the information of the business object data table to the first data warehouse, so that the first data warehouse generates the business object data table in real time; Alternatively, if fault information is received that the second data warehouse generates the business object data table offline based on the business object structured data, information of the business object data table generated in real time by the first data warehouse based on the business object structured data is obtained and displayed, and the business object data table indicated by the information of the business object data table is sent to the second data warehouse, so that the second data warehouse generates the business object data table offline.

14. A data processing device, comprising: A first acquisition module is used to acquire business object structured data loaded by an ETL tool for generating a first business object data table, wherein the business object structured data is obtained by the ETL tool performing a unified analysis on business object data acquired in real time from different data sources; A first generating module, configured to load the business object structured data into a first data warehouse, and generate the first business object data table in real time according to the business object structured data; A first import module, used to import the first business object data table into a business object delivery platform for display; The first generation module is also used for, when the first data warehouse fails to generate the first business object data table in real time, the first data warehouse obtains the second business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period, and generates the first business object data table in real time according to the second business object data table.

15. A data processing device, comprising: A second acquisition module is used to acquire business object structured data loaded by the ETL tool for generating a second business object data table within a preset time period, wherein the business object structured data is obtained by the ETL tool performing a unified analysis on the business object data acquired in real time from different data sources; A third generating module, configured to load the business object structured data into a second data warehouse, and offline generate the second business object data table according to the business object structured data; A second import module, used to import the second business object data table into the business object delivery platform for display; The third generation module is also used for, when the second data warehouse fails to generate the second business object data table offline, the second data warehouse obtains the first business object data table generated in real time by the first data warehouse according to the business object structured data, and generates the second business object data table offline according to the first business object data table.

16. A data processing device, comprising: A first receiving module, configured to receive a business object data table generation trigger operation input by a user, and display information of business object structured data used to generate a business object data table according to the trigger operation; A selection module is used to send the business object structured data selected by the selection operation to the first data warehouse and the second data warehouse respectively according to the selection operation of the user on the information of the displayed business object structured data, so that the first data warehouse generates the business object data table in real time according to the business object structured data, and the second data warehouse generates the business object data table within a preset time period offline according to the business object structured data; The selection module is further configured to obtain and display information of the business object data table generated offline by the second data warehouse according to the business object structured data within a preset time period if failure information of the first data warehouse generating the business object data table in real time according to the business object structured data is received, and send the business object data table indicated by the information of the business object data table to the first data warehouse, so that the first data warehouse generates the business object data table in real time; or, If fault information is received that the second data warehouse generates the business object data table offline based on the business object structured data, the information of the business object data table generated in real time by the first data warehouse based on the business object structured data is obtained and displayed, and the business object data table indicated by the information of the business object data table is sent to the second data warehouse, so that the second data warehouse generates the business object data table offline.

17. An electronic device, comprising: one or more processors; A computer readable medium configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1 to 6, or implement the data processing method as described in any one of claims 7 to 8, or implement the data processing method as described in any one of claims 9 to 10, or implement the data processing method as described in any one of claims 11 to 12.

18. A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any one of claims 1 to 6, or the data processing method as described in any one of claims 7 to 8, or the data processing method as described in any one of claims 9 to 10, or the data processing method as described in any one of claims 11 to 12.

Images