Data processing method, platform, computer readable storage medium and electronic device

Abstract

The present disclosure relates to a data processing method, platform, computer readable storage medium and electronic device, and relates to the technical field of big data processing. The method comprises the following steps: based on a preset standardized data interface, extracting original data to a message engine; wherein the original data comprises terminal data generated by a terminal device and / or system data generated by an application system (S110); reading the original data from the message engine and storing the original data in a distributed file system to form a fact table (S120); storing the fact table in a columnar database and obtaining target column data required for generating a target chart from the columnar database (S130); and generating the target chart according to the target column data (S140). The present disclosure improves the accuracy of data processing results.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of big data processing, and in particular, to a data processing method, a data processing platform, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] The advent of the big data era has significantly changed people's daily production and life style. With the convenience of the big data era, the business processing of traditional industries has achieved intelligentization and high efficiency.

[0003] At present, the management mode of data of various enterprises is to manage respectively by using multiple independent systems. For example, an application system management system for managing various application systems and a device management system for managing terminal devices.

[0004] However, since the terminal information and the system information exist independently, the information cannot be interconnected, which leads to that the operation conditions of the terminal and the system cannot be analyzed simultaneously according to the terminal information and the system information, or the operation conditions of the terminal and the system can only be analyzed through the terminal information and the system information respectively, and thus the accuracy of the analysis result is low. SUMMARY

[0005] According to one aspect of the present disclosure, a data processing method is provided, comprising:

[0006] extracting original data into a message engine based on a preset standardized data interface; wherein the original data comprises terminal data generated by a terminal device and / or system data generated by an application system;

[0007] reading the original data from the message engine and storing the original data into a distributed file system to form a fact table;

[0008] storing the fact table into a columnar database and obtaining target column data required for generating a target chart from the columnar database;

[0009] generating the target chart according to the target column data.

[0010] In an exemplary embodiment of the present disclosure, extracting original data into a message engine based on a preset standardized data interface comprises:

[0011] receiving the terminal data pushed by a platform where the terminal device is located based on the preset standardized data interface; wherein the terminal data comprises one or more of device state behaviors, command log data, and hardware product data of the terminal device; and

[0012] Pull the system data from the relational database at intervals of a first preset time period based on a batch integration tool and a preset standardized data interface;

[0013] Converge the terminal data and / or the system data into the message engine.

[0014] In an exemplary embodiment of the present disclosure, converging the terminal data and / or the system data into the message engine comprises:

[0015] Converging the terminal data and / or the system data into the message engine under a topic corresponding to a data event type of the terminal data and / or the system data.

[0016] In an exemplary embodiment of the present disclosure, the fact table comprises a detailed fact table.

[0017] Wherein, reading the raw data from the message engine and storing the raw data into a distributed file system to form a fact table comprises:

[0018] Reading the raw data from the message engine and storing the raw data into the distributed file system under a date corresponding to a data occurrence timestamp of the raw data according to the data occurrence timestamp of the raw data;

[0019] Reading the raw data from the distributed file system and constructing a raw data layer according to the raw data;

[0020] Calculating a subject domain of the raw data according to the raw data included in the raw data layer and a data dictionary corresponding to the raw data;

[0021] Summarizing the raw data according to the subject domain to generate the detailed fact table.

[0022] In an exemplary embodiment of the present disclosure, reading the raw data from the distributed file system and constructing a raw data layer according to the raw data comprises:

[0023] Creating a raw data table comprising data source, data reporter, data version number, data sending location, data occurrence timestamp, data event type and data detailed information;

[0024] Reading the raw data according to a storage location of the raw data in the distributed file system;

[0025] Writing the raw data into a position corresponding to a current date of a data occurrence timestamp of the raw data in the raw data table according to the current date of the data occurrence timestamp of the raw data to obtain the raw data layer.

[0026] In an example embodiment of the present disclosure, a subject domain of the original data is calculated according to the original data included in the original data layer and a data dictionary corresponding to the original data, including:

[0027] A data dictionary corresponding to the original data is obtained, and a data service corresponding to the original data is determined according to the data dictionary.

[0028] A subject domain of the original data is determined according to the data service; wherein the subject domain includes one or more of a terminal domain, a system domain, and a user domain.

[0029] In an example embodiment of the present disclosure, a data dictionary corresponding to the original data is obtained, including:

[0030] When the original data is system data, metadata corresponding to the original data is pulled as the data dictionary.

[0031] When the original data is terminal data, the data dictionary is created according to a directory included in the metadata.

[0032] In an example embodiment of the present disclosure, the original data is summarized according to the subject domain to generate the detailed fact table, including:

[0033] When the original data is terminal data, the original data is summarized into a report with a subject domain of a terminal domain to generate a detailed fact table corresponding to the terminal domain.

[0034] When the original data is system data, the original data is summarized into a report with a subject domain of a system domain to generate a detailed fact table corresponding to the system domain.

[0035] In an example embodiment of the present disclosure, the fact table further includes a periodic snapshot fact table.

[0036] The data processing method further includes:

[0037] The detailed fact table is aggregated based on a preset aggregation rule to obtain the periodic snapshot fact table.

[0038] The preset aggregation rule includes aggregation in a preset time interval.

[0039] In an example embodiment of the present disclosure, the detailed fact table is aggregated based on a preset aggregation rule to obtain the periodic snapshot fact table, including:

[0040] According to attribute information of terminal equipment included in the terminal data and attribute information of application systems included in system data, an attribute dimension table is generated;

[0041] According to common attributes between terminal equipment and the application systems included in the attribute dimension table, an association relationship between a detailed fact table corresponding to a terminal domain and a detailed fact table corresponding to a system domain is established;

[0042] Based on the association relationship, the detailed fact table corresponding to the terminal domain and the detailed fact table corresponding to the system domain are aggregated in a preset time interval to obtain the periodic snapshot fact table.

[0043] According to one aspect of the present disclosure, a data processing platform is provided, comprising:

[0044] A data aggregation layer, based on a preset standardized data interface, extracts raw data to a message engine;

[0045] A data storage layer, connected with the data aggregation layer, is configured to read the raw data from the message engine and store the raw data to a distributed file system;

[0046] A data processing layer, connected with the data storage layer, is configured to read the raw data from the distributed file system and generate a fact table according to the raw data; and

[0047] Store the fact table to a columnar database and obtain target column data required for generating a target chart from the columnar database; and

[0048] Generate the target chart according to the target column data.

[0049] In an exemplary embodiment of the present disclosure, the data processing layer comprises a raw data layer, a unified data warehouse layer, an application data layer, a label data layer and a dimension table;

[0050] The unified data warehouse layer is connected with the raw data layer, the application data layer and the label data layer are respectively connected with the unified data warehouse layer, and the dimension table is included in the unified data warehouse layer;

[0051] The raw data layer is configured to read the raw data from a file storage system and store the raw data;

[0052] The unified data warehouse layer is configured to calculate a subject domain of the raw data according to the raw data included in the raw data layer and a data dictionary corresponding to the raw data; and

[0053] aggregate the original data according to the subject domain to generate a detailed fact table; and

[0054] generate an attribute dimension table according to attribute information of terminal equipment included in terminal data included in the original data and attribute information of application systems included in system data included in the original data; and

[0055] generate a periodic snapshot fact table according to the detailed fact table and the attribute dimension table;

[0056] The application data layer is configured to acquire target column data required for generating a target chart from the column database, and generate the target chart according to the target column data.

[0057] The label data layer is configured to identify terminal equipment and application systems included in the original data.

[0058] According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the data processing method according to any one of the preceding aspects.

[0059] According to an aspect of the present disclosure, there is provided an electronic device, comprising:

[0060] a processor; and

[0061] a memory configured to store executable instructions of the processor;

[0062] The processor is configured to implement the data processing method according to any one of the preceding aspects by executing the executable instructions.

[0063] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is apparent that the accompanying drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0065] Figure 1 a flow chart of a data processing method according to an example embodiment of the present disclosure is schematically shown.

[0066] Figure 2 a structural block diagram of a data processing platform according to an example embodiment of the present disclosure is schematically shown.

[0067] Figure 3 A structural block diagram of a data processing layer according to an example embodiment of the present application is schematically shown.

[0068] Figure 4 An example diagram of a standard data format according to an example embodiment of the present application is schematically shown.

[0069] Figure 5 A method flowchart of reading the raw data from the message engine and storing the raw data into a distributed file system to form a fact table according to an example embodiment of the present application is schematically shown.

[0070] Figure 6 A method flowchart of reading the raw data from the distributed file system and constructing a raw data layer according to the raw data according to an example embodiment of the present application is schematically shown.

[0071] Figure 7 An example diagram of a subject domain according to an example embodiment of the present application is schematically shown.

[0072] Figure 8 An example diagram of a data dictionary according to an example embodiment of the present application is schematically shown.

[0073] Figure 9 A method flowchart of aggregating the detailed fact table based on preset aggregation rules to obtain the periodic snapshot fact table according to an example embodiment of the present application is schematically shown.

[0074] Figure 10 A flowchart of another data processing method according to an example embodiment of the present application is schematically shown.

[0075] Figure 11 A block diagram of a data processing apparatus according to an example embodiment of the present application is schematically shown.

[0076] Figure 12 An electronic device for implementing the above data processing method according to an example embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0077] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example embodiments to those skilled in the art. Features, structures, or characteristics described in connection with one example embodiment can be combined in any suitable manner with features, structures or characteristics of other example embodiments.

[0078] In addition, the accompanying drawings are only schematic and are not necessarily drawn to scale. The same reference numerals in different drawings denote the same or similar functionality throughout the several drawings; and

[0079] In the present example embodiment, a data processing method is first provided, which can be run on a server, a server cluster, a cloud server, or the like where a data platform is located. Of course, those skilled in the art can also run the method of the present application on other platforms according to requirements, which is not specially limited in the present example embodiment. Referring to Figure 1 As shown in the figure, the data processing method can include the following steps:

[0080] Step S110. Based on a preset standardized data interface, raw data is extracted into a message engine; wherein the raw data includes terminal data generated by a terminal device and / or system data generated by an application system;

[0081] Step S120. The raw data is read from the message engine and stored into a distributed file system to form a fact table;

[0082] Step S130. The fact table is stored into a columnar database, and target column data required for generating a target chart is obtained from the columnar database;

[0083] Step S140. The target chart is generated according to the target column data.

[0084] In the above data processing method, on the one hand, raw data is extracted into a message engine based on a preset standardized data interface; then, the raw data is read from the message engine and stored in a distributed file system to form a fact table; next, the fact table is stored in a columnar database, and the target column data required to generate the target chart is obtained from the columnar database; finally, the target chart is generated based on the target column data. Since system data and terminal data can be extracted into the message engine simultaneously to generate the target chart, this solves the problem in the prior art where terminal information and system information exist independently, making it impossible to achieve information interconnection and interoperability, thus preventing the simultaneous processing of terminal and system data based on both terminal and system information. On the one hand, it addresses the issue of analyzing the operational status of devices; on the other hand, by obtaining the target column data required to generate the target chart from the columnar database, and generating the target chart based on the target column data, the operational status of terminal devices and application systems can be analyzed based on the target chart. This avoids the problem of low accuracy in analysis results caused by only being able to analyze the operational status of terminals based on terminal information or only being able to analyze the operational status of systems based on system information. Furthermore, by extracting raw data into the message engine based on a preset standardized data interface, it avoids the problem of data processing failure or incorrect processing results due to inconsistent standards of current data, further improving the accuracy of processing results.

[0085] The following will provide a detailed explanation and description of each step described in the data processing method of this disclosure, in conjunction with the accompanying drawings.

[0086] First, the terms used in this disclosure will be explained and clarified.

[0087] A data standardization interface is an interface that supports data collection from both software (application systems) and hardware (terminal devices). Application systems include web-based systems and application software, while hardware includes interactive terminals such as touch-screen tablets, drawing tablets, and writing tablets, as well as non-interactive terminals such as televisions and communication terminals. It also supports the collection of offline data sources and real-time data streams. Offline data sources include relational databases such as MySQL and PostgreSQL used in business products, while real-time data streams include IoT (Internet of Things) platforms that collect data from terminal data points and real-time data streams from system data points.

[0088] HDFS, or Hadoop Distributed File System, is a highly fault-tolerant system suitable for deployment on inexpensive machines, making it ideal for persistent storage of very large datasets. HDFS also provides high-throughput data access, making it well-suited for applications involving large datasets.

[0089] Data assets are data resources, such as documents and electronic data, owned or controlled by an enterprise and capable of generating future economic benefits for the enterprise. Characteristics of data assets include: comprehensive data coverage, clear hierarchical structure, accurate and consistent data, improved performance, reduced costs, and ease of use. Data assets are the core product of a data platform, connecting data from all products across all business lines within an enterprise. This allows data to be processed only once, enabling data reuse and sharing, eliminating the previous siloed development and storage practices of each business unit, and reducing development costs and human and material resources.

[0090] Webhook is an API concept and one of the usage paradigms of microservice APIs. It means that the front end does not actively send messages, but is completely pushed by the back end. When the terminal or system data points generate data, the data will be actively distributed and pushed to the data receiving interface.

[0091] Kafka, a message engine, supports cluster mode, enabling multiple identical backups to provide redundancy and ensure high system availability. It can persistently store data (typically set to 7 days), meaning data sent to Kafka within 7 days will not be deleted and will be stored on disk. Furthermore, Kafka's core architecture involves producers sending messages to Kafka and consumers reading messages from the Kafka server. It supports both of the two most common message engine paradigms: message queues and publish / subscribe.

[0092] ClickHouse is a column-oriented DBMS (Database Management System), meaning it can store individual column values ​​separately. It allows tables and databases to be created, data loaded, and queries run at runtime without requiring server reconfiguration or restart. It improves performance through data compression, supports disk storage for increased data storage capacity, and supports real-time data updates, with data continuously added to tables without lock-free processing. Its biggest advantages are fast query speeds and large data volumes (PB-level). It supports SQL-like queries and geographically distributed database replication, making it ideal for data source databases supporting data service applications.

[0093] Real-time data, including but not limited to: through the terminal embedded data collection method to collect the Internet of Things platform real-time data stream to collect real-time data through the software embedded data collection method; Real-time data is collected through the distribution of data standardization interface; The data receiving interface uses a reverse application programming interface, namely Webhook;

[0094] Offline data, including but not limited to: a certain amount of data accumulated after a period of time, the data stored in the relational database mysql, PostgreSQL frequently used in business products can be classified as offline data. Offline data can use batch pulling tools to update in batches, such as pulling the entire database data once a day, or pulling the entire database data to the hour level. Batch pulling tools can use log collection tools flume or open source stream processing framework Flink to implement batch pulling of offline data.

[0095] Offline development refers to batch computing. Offline computing is mainly used for high-latency processing of batch data, such as processing of offline data warehouse, cleaning and mining of large-scale data, etc. MapReduce, Spark and other computing frameworks can be used.

[0096] Real-time development refers to stream computing. Stream computing is suitable for processing and application of data with strong timeliness requirements. Common monitoring and alarm scenarios include real-time analysis of network events. Common computing frameworks include Flink, Spark Streaming and Storm.

[0097] Detail fact table, used to record transaction-level facts, save atomic data, the granularity of data is usually one record per transaction, the data of detail fact table is inserted and no longer changed, and the update method is incremental update. Among them, the fact refers to the actual use of application system and terminal device, for example, the function that the application system can support, the user information included in the application system, the actual attribute of the terminal device, the file format that the terminal device can support to play, the online and offline situation of the terminal device, etc.

[0098] A summary fact table, also known as a periodic snapshot fact table, is a fact table formed by aggregating detailed facts. This includes periodic snapshot fact tables that record facts at regular, predictable time intervals. In a periodic snapshot fact table, snapshots are generated at regular, predictable time intervals to record facts. Each row represents a record for a specific time period, and the recorded facts are aggregated fact values ​​or state measures within that time period. The content of a periodic snapshot fact table is generally generated only after the expressed time period ends, and the data is generally not changed after recording. Its update method is incremental. Periodic snapshot fact tables are generally aggregated on top of transaction fact tables, but they have fewer dimensions and coarser granularity than transaction fact tables.

[0099] Secondly, the purpose of this invention will be explained and described. Specifically, under traditional IT infrastructure, most enterprise information systems are independently purchased or built, making information interconnection impossible and resulting in multiple data silos within the enterprise. The development of the Internet and mobile Internet has brought about many new business models, generating data that cannot be interoperated with data from traditional models, further exacerbating the data silo problem; at the same time, data scattered across various silos cannot effectively support enterprise business decisions or cope with rapidly changing business needs.

[0100] Therefore, enterprises need a mechanism to integrate new and old models, consolidate data scattered across various isolated platforms, and quickly form data service capabilities to support business decision-making and refined operations. This mechanism is the data middle platform.

[0101] Specifically, a data platform refers to the use of data technology to collect, compute, store, and process massive amounts of data, while unifying data standards and definitions. It also includes the model services, algorithm services, organizational structure, processes, standards, specifications, and management systems required for building the data platform. After unifying the data, the data platform creates standardized data, which is then stored to form data assets. Through data mining and analysis tools, it enables data service capabilities, ultimately providing efficient services to enterprises and customers.

[0102] Furthermore, the data processing platform disclosed herein (i.e., the data middle platform mentioned above) will be explained and described.

[0103] For details, please refer to Figure 2 As shown, the data processing platform may include a standardized data interface 210, a data aggregation layer 220, a data storage layer 230, and a data processing layer 240; it may also include a data service layer 250. Wherein:

[0104] The data aggregation layer 220 is connected with the standardized data interface 210, and can be used to extract original data to a message engine based on a preset standardized data interface; the data storage layer 230 is connected with the data aggregation layer, and is used to read the original data from the message engine and store the original data into a distributed file system; the data processing layer 240 is connected with the data storage layer, and is used to read the original data from the distributed file system, generate a fact table according to the original data, store the fact table into a columnar database, acquire target column data required for generating a target chart from the columnar database, and generate the target chart according to the target column data; and the data service layer 250 is connected with the data processing layer, and is used to analyze operation conditions of terminal equipment and application systems included in the target chart according to the target chart.

[0105] Further, referring to FIG. 2, Figure 3 As shown in FIG. 2, the data processing layer 240 can include an original data layer ODS (Operational Data Store) 301, a unified data warehouse layer DWD / DWS (Data Warehouse Detail / Data Warehouse service) 302, an application data layer ADS (Application Data Store) 303, a tag data layer TDM (Tag Data Model) 304, and a dimension table DIM (Dim data) 305; the unified data warehouse layer is connected with the original data layer, the application data layer and the tag data layer are respectively connected with the unified data warehouse layer, and the dimension table is included in the unified data warehouse layer; that is, data of a general DWD / DWS data layer is sourced from the ODS layer, and data of a TDM / ADS data layer is sourced from the DWD / DWS data layer; according to actual conditions, data assets can be composed of all or part of data layers.

[0106] The unified data warehouse layer is used to calculate a subject domain of the original data according to the original data and a data dictionary corresponding to the original data included in the original data layer, aggregate the original data according to the subject domain to generate a detailed fact table, generate an attribute dimension table according to attribute information of terminal equipment included in terminal data and attribute information of application systems included in system data included in the original data, and generate a periodic snapshot fact table according to the detailed fact table and the attribute dimension table; the application data layer is used to acquire target column data required for generating a target chart from the columnar database, generate the target chart according to the target column data; and the tag data layer is used to identify terminal equipment and application systems included in the original data.

[0107] The construction method of the data platform is explained and described as follows. Specifically, the construction method of the data platform mainly includes five processes, i.e., a data collection step, a data aggregation step, a data storage step, a data development step, and a data analysis step.

[0108] The data collection is implemented by using a general data receiving interface (standardized data interface) service, and the data sources include real-time data and offline data on each terminal device and application system. The data aggregation is implemented by using a message engine, which can achieve the purpose of traffic peak clipping and the role of data transmission channel. The data storage implements persistent storage of real-time data and offline data, which can meet the requirements of back calculation or data supplement on historical data. The data development includes batch calculation and stream calculation. The batch calculation is suitable for high-latency processing scenarios of batch data, and the stream calculation is suitable for data processing and application with strong timeliness requirements. The data analysis implements data value mining according to data assets and reversely enables the business. Through data modeling, a global tag system of data objects is formed to provide data for business application.

[0109] The steps S110-S140 will be explained and described in combination with the data platform as described above.

[0110] In step S110, the original data is extracted into the message engine based on the preset standardized data interface. The original data includes terminal data generated by a terminal device and / or system data generated by an application system.

[0111] In the example embodiment, reference is made to Figure 4As shown, the attribute information included in the standardized data interface can include data source (from system), data reporter, data version number, data sending location, data event time, data event type, and data detailed information (arguments); at the same time, the data detailed information can include project name, first event parameter corresponding to the project name, project note, second event parameter corresponding to the project note, project address, third event parameter corresponding to the project address, number of added files, fourth event parameter corresponding to the number of added files, submission note, and fifth event parameter corresponding to the submission note, and the like; wherein, taking the project name as EPD_Transform (the first event parameter as an example), the second event parameter corresponding thereto is EPD intelligent color filter, the third event parameter is: git@*****: abid / image_process_group / EPD_transform.git, the fourth event parameter is 3, and the fifth event parameter is to fix slow bug of response time. It should be noted here that the standardized data receiving interface is universal, which can be reused for collecting real-time data streams from various sources. The universal data receiving interface first needs to define a universal standard data format, which can be in json format or xml format, but is not limited to these two formats.

[0112] Specifically, based on the preset standardized data interface, the original data is extracted into the message engine, which can include: first, based on the preset standardized data interface, receiving the terminal data pushed by the platform where the terminal device is located; wherein, the terminal data includes one or more of device state behavior, command log data and hardware product data of the terminal device; second, based on the batch integration tool and the preset standardized data interface, pulling the system data from the relational database at intervals of a first preset time period; and finally, converging the terminal data and / or the system data into the message engine.

[0113] The device state behavior represents the online state and offline state of the device, the command log data can include the commands (such as switch commands or play commands, etc.) received by the terminal device, and the hardware product data can include the attribute information of the terminal device itself, such as supported resolution and device size, etc. Each hardware product data can be collected by sensors, cameras, etc. For example, sensors are used, a reliable transmission, low-latency MQTT (Message Queuing Telemetry Transport) protocol is used to build an Internet of Things framework, data communication between the device end and the service end is realized, a cluster is built to support a large number of connected terminals, and the device state behavior and command log data of the terminal can be actively collected, based on which terminal data can be obtained.

[0114] Meanwhile, since each application system interacts with users every day to generate a large amount of data, these data can be collected by using a burying point technology to realize real-time data collection. Meanwhile, a part of data of each application system is stored in a database through a backend interface to form offline data.

[0115] Further, after receiving the terminal data and the system data through the standardized data interface, the obtained raw data is processed into standardized data including data source, data reporter, data version number, data sending location, data occurrence timestamp, data event type, and data detailed information. After receiving the terminal data and / or pulling the system data, the terminal data and / or the system data can be converged into the message engine. Specifically, the terminal data and / or the system data can be converged into the message engine under a topic corresponding to the data event type of the terminal data and / or the system data.

[0116] Specifically, the message engine can play a role in traffic peak shaving, and also play a role in message queuing and message publishing and subscribing, and can temporarily store data. The message engine can be implemented by using Kafka. Further, the real-time data stream collected through the general data receiving interface (standardized data interface) is transmitted into the message engine, and the data in the offline data source obtained by batch pulling is also transmitted into the message engine. The general data receiving interface and the batch pulling tool are equivalent to the message producers of Kafka, which continuously transmit the collected data, real-time data or batch-pulled overall offline data, into the specified Topic. Moreover, the Topic topic in Kafka is only a logical concept, representing a class of messages, and can be regarded as a place where messages are sent. The producer transmits data into Kafka, and each producer needs to specify the Topic to which the message is sent. The same Topic can be subscribed by multiple consumers.

[0117] For example, the product terminal of an enterprise is to use the Internet of Things platform to collect real-time data, log records, command results, message states, and other real-time data streams. The data collected by the Internet of Things platform can be distributed to a general data receiving interface and then transmitted to a specified Topic in Kafka to realize the convergence of real-time data streams. The source data is stored in a file storage system of a big data platform under a specified directory by consuming Kafka. The file storage system is a distributed file system of Hadoop.

[0118] In step S120, the raw data is read from the message engine and stored in a distributed file system to form a fact table.

[0119] In the example embodiment, referring to FIG. 5, reading the raw data from the message engine and storing the raw data in a distributed file system to form a fact table can include steps S510-S540. In the steps, the raw data is read from the message engine and stored in the distributed file system according to a data occurrence timestamp of the raw data. Figure 5

[0120] In step S510, the raw data is read from the message engine and stored in the distributed file system according to a data occurrence timestamp of the raw data.

[0121] Specifically, the data persistence storage can be implemented by using a Hadoop distributed file system (HDFS) of a big data platform. Further, the raw data in the message engine Kafka is read by using a Kafka topic consumer function and stored in the HDFS. Regardless of system data or terminal data, after being collected and transmitted to Kafka, there is a corresponding Topic. When reading, a consumer is started for each Topic, and the source data in a Topic is stored in a corresponding path directory of the HDFS in the big data cluster to realize the persistence storage of the raw data. The path directory includes a date.

[0122] At the same time, since the raw data is extracted through a standardized data interface, it has a unified data format. By this method, the subsequent offline development can be facilitated, so that regardless of the initial data format of each business, after being unified, a program can be used for processing, reducing the workload and avoiding the understanding of business data metadata by developers.

[0123] In step S520, the raw data is read from the distributed file system and an original data layer is constructed according to the raw data.

[0124] In the example embodiment, referring to FIG. 5, reading the raw data from the message engine and storing the raw data in a distributed file system to form a fact table can include steps S510-S540. In the steps, the raw data is read from the message engine and stored in the distributed file system according to a data occurrence timestamp of the raw data. Figure 6 ​As shown, reading the current data from the distributed file system and constructing a raw data layer according to the current data can include steps S610-S630. Wherein:

[0125] In step S610, a raw data table including data source, data reporter, data version number, data sending location, data occurrence timestamp, data event type and data details is created;

[0126] In step S620, the raw data is read according to the storage location of the raw data in the distributed file system;

[0127] In step S630, the raw data is written into the raw data table corresponding to the current date according to the data occurrence timestamp of the raw data, and the raw data layer is obtained.

[0128] In the following, steps S610-S630 will be explained and described. Specifically, to construct the ODS layer of the data warehouse, the Hive component in the big data platform can be used. First, the table creation statement of Hive needs to be created. The corresponding storage location is the storage location of the corresponding source data table under the corresponding business directory on HDFS. Specifically, it can be as shown in Table 1:

[0129] Table 1

[0130]

[0131]

[0132] It should be noted here that the system data and the terminal data are constructed in the same way. After creating the table creation statement of Hive, initialization is performed, i.e. adding the partition of the earliest day of data to the Hive table. Thereafter, the partition needs to be brushed every day, i.e. adding the partition of the previous day to the Hive table. In this way, the data on HDFS (Hadoop Distributed FileSystem, Hadoop Distributed File System) will correspondingly enter the Hive table.

[0133] In step S530, the subject domain of the raw data is calculated according to the raw data included in the raw data layer and the data dictionary corresponding to the raw data.

[0134] In the example embodiment, first, a data dictionary corresponding to the original data is acquired, and a data service corresponding to the original data is determined according to the data dictionary; second, a subject domain of the original data is determined according to the data service; wherein the subject domain includes a terminal domain, a system domain, a user domain, and the like. Wherein, acquiring the data dictionary corresponding to the original data can specifically include: when the original data is system data, pulling metadata corresponding to the original data as the data dictionary; when the original data is terminal data, creating the data dictionary according to a directory included in the metadata.

[0135] Specifically, the premise of determining the report indicators of each service is to set the subject domain of the service. Specifically, the subject domain can include a first-level subject domain and a second-level subject domain, and the like; wherein the first-level subject domain can be a terminal domain, a system domain, a user domain, and the like, and the second-level subject domain can be customized by a data warehouse developer, such as a DMA domain, a sensor domain, and a log domain under the terminal domain; then, the indicators are specified under each subject domain; specifically, taking the terminal domain as an example, as shown in Figure 7 , the specific content includes a serial number (id) 701, a subject domain 702, an indicator name 703, an indicator meaning 704, an implementation chart type 705, and a creator 706, and the like; this process needs to be communicated with the business-related personnel for many times, and the final indicator version can be determined after many iterations and updates.

[0136] Further, the data dictionary of the mysql database of the system can pull the metadata of the mysql database by the docker image of Flink, and store it in the PostgreSQL database of the big data platform, and update it in a zip table manner once a day; wherein each field of each table in the database is a row of data in the PostgreSQL database table, and includes a creation time and an update time, if the information of this field (for example, field type / whether it can be empty / comment) changes, a new data of this field is created, the creation time and the update time are the same as the pulling time of the image, the update time of the original data row of this field is the image pulling time, and the life cycle of the original data row is represented as the current time point. For example, as shown in the following table; at the same time, since the multiple Topics of the Internet of Things platform do not have system-stored metadata information, the data dictionary needs to be manually constructed, and the corresponding data dictionary can be constructed according to the directory of the data dictionary of the mysql database, which can be specifically referred to Figure 8 .

[0137] For example, in Figure 8In the shown data dictionary, the corresponding directory can include: data source (from_system) 801, database (database) 802, table name (table_name) 803, field name (column_name) 804, field type (column_type) 805, whether the type is nullable (is_nullable) 806, comment (comment) 807, and the like. Taking the signal sending system as an example, the data source can be xinfa, the database can be screenosserver, the table name can be device, the field type can include create_time, media_volume, run_time, and the like, the field type can include timestamp, int(10), varchar(255), and the comment can include create_time, media_volume, and device run time, and the like.

[0138] In step S540, the original data is summarized according to the subject domain to generate the detailed fact table.

[0139] In the example embodiment, first of all, it needs to be pointed out that the process of summarizing the original data according to the subject domain and then generating the detailed fact table can be realized in an offline development and real-time development manner. For example, batch calculation in data development of the data center is applicable to calculating the original data, and the calculation results (detailed fact table) available for data service are stored in the columnar database ClickHouse. At the same time, batch calculation can use the Spark framework, and the intermediate data warehouse can use the hive tool. Stream calculation is applicable to using the Flink real-time processing tool to consume data from Kafka to directly perform data cleaning, aggregation, and the like.

[0140] Further, in the example embodiment, first, the original data is summarized according to the subject domain to generate the detailed fact table, which can specifically include: when the original data is terminal data, the original data is summarized into a report with a subject domain of terminal domain to generate a detailed fact table corresponding to the terminal domain; when the original data is system data, the original data is summarized into a report with a subject domain of system domain to generate a detailed fact table corresponding to the system domain.

[0141] Further, when the detailed fact table is obtained, the data processing method can further include: aggregating the detailed fact table based on a preset aggregation rule to obtain the periodic snapshot fact table; wherein the preset aggregation rule includes aggregating in a preset time interval manner.

[0142] Further, referring to Figure 9As shown, the aggregating the detail fact table based on the preset aggregation rule to obtain the periodic snapshot fact table can include steps S910-S930. Wherein:

[0143] In step S910, according to the attribute information of the terminal device included in the terminal data and the attribute information of the application system included in the system data, an attribute dimension table is generated;

[0144] In step S920, according to the common attribute between the terminal device and the application system included in the attribute dimension table, an association relationship between the detail fact table corresponding to the terminal domain and the detail fact table corresponding to the system domain is established;

[0145] In step S930, based on the association relationship, the detail fact table corresponding to the terminal domain and the detail fact table corresponding to the system domain are aggregated in a preset time interval to obtain the periodic snapshot fact table.

[0146] In the following, the specific implementation process of S540 and steps S910-S930 (i.e. the specific construction process of DWD / DWS layer) will be further explained and described. Specifically, the data dictionary of the business source database understands the data meaning of the business, and combined with the determined report indicators, the DWD / DWS layer of the data warehouse can be constructed. The DWD layer data warehouse adopts the dimension modeling data warehouse modeling method, which organizes data with current fact table and dimension table, that is, constructs the fact wide table of each entity of the business, and constructs the dimension table of the entity (application system and terminal device). At the same time, since the subject domain has been determined, when the current data is terminal data, it is directly summarized into the report with the subject domain as the terminal domain to generate the detail fact table corresponding to the terminal domain; when the current data is system data, it is directly summarized into the report with the subject domain as the system domain to generate the detail fact table corresponding to the system domain.

[0147] Further, in order to generate the periodic snapshot fact table, that is, to obtain the association between the terminal data and the system data, a dimension table also needs to be generated. Specifically, an attribute dimension table can be generated according to the attribute information of the terminal device included in the terminal data and the attribute information of the application system included in the system data; the attribute dimension table can include the user's access mac address, the user's login name (such as the user's mobile phone number or ID card number), the user's login time, and the attribute information of the application system itself (size, supported installed device, and version inherent attributes), the mac address of the terminal device, the online time of the terminal device, the offline time of the terminal device, the specific playing content of the terminal device, the identification information of the user who purchased the terminal device (such as the user's mobile phone number or ID card number), and the attribute information of the terminal device itself (size, resolution, and other inherent attributes), and the like; after obtaining the dimension table, the association between the detailed fact table corresponding to the terminal domain and the detailed fact table corresponding to the system domain can be established based on the dimension table (such as the user's mobile phone number or ID card number); finally, based on the association, the periodic snapshot fact table is obtained, so as to analyze the operation of the terminal device and the application system.

[0148] That is, in order to count the common users of the terminal device and the application system, the Spark framework can be used to read and store the user data in the source data of all business products (terminal device and application system) into Hive to form an ODS data layer; after detailed aggregation of all user data, the selected user multiple dimension attributes are used to judge the common users to form a user theme domain DWD data layer, these attributes are used to determine the uniqueness of the user, and the attributes can use the user's mobile phone number or ID card number, etc., and the DWD data layer is also stored in the hive warehouse; some aggregation indexes are calculated, such as the total number of users of the product in each day, the number of new users per day, etc., to form a DWS data layer. The ODS / DWD / DWS data layers together form a data asset, and after obtaining the data asset, data services can be quickly and conveniently expanded, such as operation analysis, which uses visual report to display the results; or data analysis, which discovers new knowledge through rule algorithm, classification algorithm, clustering algorithm, etc.

[0149] It needs to be further supplemented here that the tables of the DWD warehouse are divided into fact tables and dimension tables, and the updating process of the incremental data of the fact table is: if the source data in the ODS layer is updated, that is, the update_time field in the data table is updated to the time stamp of the day when the data storage task to the big data platform is executed, a new data is added in the fact table, and the fact data in the ODS layer corresponding table is added to the fact table; the updating process of the dimension table is: the dimension table updates the attribute corresponding to the existing entity value, and a new entity value is added to store a data in the dimension table.

[0150] And, the current ODS / DWD / DWS layer data is stored in Hive, and the DWD / DWS data warehouse is also implemented by using SQL calculation in Hive, and the calculation task is also a daily timing task, which is set to start at 6 o'clock in the morning every day. In addition to being consistent with the data warehouse construction process of the system, the data warehouse construction process of the real-time data stream of the Internet of Things platform also has a data cleaning task, which is also a batch processing task using the Spark framework and a daily timing task, which is set to start at 5 o'clock in the morning every day.

[0151] In step S130, the fact table is stored in the columnar database, and target column data required for generating a target chart is obtained from the columnar database.

[0152] Specifically, after obtaining the above detailed fact table and the periodic snapshot fact table, the detailed fact table and the periodic snapshot fact table can be stored in a columnar database (ClickHouse, of course, it can also be HBase) to provide fast data query for data services. At the same time, the DWD / DWS layer tables used for report making in Hive are transmitted to the columnar database ClickHouse, and the database connected by the business intelligence tool for report making is ClickHouse; the data synchronization task of transmitting the Hive table data to ClickHouse is also a timing task, which is set to start at 7 o'clock in the morning every day; wherein, the Jar package for realizing data synchronization is reusable for multiple data sources, and by modifying the configuration file, the task of synchronizing multiple tables can be realized.

[0153] In step S140, the target chart is generated according to the target column data.

[0154] In the example embodiment, the target chart can be generated based on the business intelligence tool according to the pulled target column data; then, the operation situation of the terminal device and the application system included in the target chart is analyzed according to the target chart; specifically, the use quantity of the terminal device by the user corresponding to the application system can be analyzed according to the target chart; the play quantity of the video data generated by using the application system on the terminal device can also be analyzed according to the target chart; and the online situation of the application system and the terminal device can also be analyzed.

[0155] Of course, the use quantity of the terminal device by the user in the future time period is also predicted according to the analysis result of the use quantity of the terminal device by the user corresponding to the application system, and the use quantity of the terminal device by the region where the user is located in the future time period is also predicted.

[0156] For example, the purchase of a certain enterprise to one or more terminal devices can be analyzed according to the target chart, the playing of some advertisements or videos on the terminal device can be analyzed according to the target chart, the online or offline times of a certain terminal device within a certain time can be analyzed, and whether the terminal device has the problem of falling off can be analyzed, and so on, or the use of a certain user to the application system can be analyzed, and then the user who needs to be focused on can be determined according to the analysis result; of course, the use quantity of the terminal device can also be predicted, and then the production quantity of the terminal device can be determined, and so on.

[0157] Of course, the data query service, the data prediction service, the user portrait, the recommendation service and the like can also be realized according to the target chart, and the data service can be expanded according to the business demand. The data service is also universal, and different source data of different businesses can be directly applied to the existing data service after the construction of the data asset. A large amount of work of the developer is saved, the data is returned to the business, the guidance is provided for the business optimization, the suggestion is provided for the company decision, and the new business and product service can be quickly promoted.

[0158] Hereinafter, the specific implementation process of the data processing method of the present disclosure is explained and described by taking a real information sending business as an example. The information sending business includes an information publishing system and a terminal. The information publishing system is composed of a server, a network, a player and a display device, and the information of the server is sent to the player through the network (wide area network / local area network / private network are applicable, including wireless network), and then the player combines audio and video, picture, text and the like (including playing position and playing content and the like) to form an audio and video file in the form of audio and video input which can be received by the liquid crystal television and the like display device to play, so that a set of network transmission of all server information to the terminal is formed. The terminal, i.e. the display device, includes a super large size terminal (98 inches / 110 inches), a picture screen terminal, a vision beauty box and the like. Specifically, referring to FIG. 1, the data processing method can include the following steps: Figure 10 The data processing method can include the following steps:

[0159] Step S1010, obtain the data of the system and the terminal on the production environment. Specifically, the database of the information publishing system is mysql, deployed in the cloud, and the system operation data and the content (materials / programs / plans) on the system of the customer are stored in the database. The data of the terminal is collected in real time through the Internet of Things platform, including the device state behavior (heartbeat data, etc.), command log data (command results are divided into info / warn / error three types), and hardware product data (including the latitude and longitude data and temperature of the terminal). The data center uses a batch integrated extraction-transformation-loading tool to obtain the data of the system in an active pull mode. The implementation is to use the Flink framework combined with the docker image technology to realize the pulling of the full data of the mysql database by day through a Flink docker image. The cloud mysql database of the production environment only needs to provide a readable account, and the full data of the mysql database is pulled by day, which does not put pressure on the production environment. The data pulled by the Flink docker image is transmitted to the Kafka cluster, and the Kafka cluster needs to be set in advance with a corresponding Topic, such as the Topic named xinfa-ab-mysql. The data of the terminal is collected in real time, and the push mode is used to obtain the terminal data. After the Internet of Things platform collects the real-time data stream of the terminal in real time, it is pushed to the data receiving interface. The data receiving interface obtains the data and transmits the data to a Topic on the Kafka cluster in real time. The Topic also needs to be set in advance, such as the Topic named iot-ab-api.

[0160] Step S1020, the data of the system and the terminal on the production environment is stored to the big data platform. Specifically, multiple Flink docker images are used to store the data on the Topic in the Kafka cluster into HDFS, and the data is stored by business as the total directory and by day as the partition. The information system uses a mysql relational database, and the pulled data is converted into a row of data which is a json string. The json string is a one-dimensional string, and the data on the Topic in Kafka can be directly read and stored into HDFS. The Internet of Things platform is designed with multiple Topics, and the formats of each Topic are different, including one-dimensional json string format and multi-dimensional nested format data. The data sources of multiple Topics of the Internet of Things platform are transmitted to the same Topic in the Kafka cluster. When reading the source data of the Internet of Things platform on the Kafka cluster, data cleaning is needed to flatten the multi-dimensional nested json string into a one-dimensional json string, and then store it into the HDFS storage location with business as the total directory and by day as the partition. The task of pulling and storing data to the big data platform is set as a daily scheduled task, which is executed from 23:00 every day. Under normal circumstances, the execution will be completed before 24:00.

[0161] In step S1030, data is acquired from the big data platform, and detailed fact tables are calculated from the data, and periodic snapshot fact tables are calculated according to the detailed fact tables. The specific calculation process is consistent with the foregoing, and will not be described here.

[0162] In step S1040, the implementation detail table and the periodic snapshot fact table are stored in the columnar database, and the required column data is pulled from the columnar database to generate a report.

[0163] In step S1050, the operation of the system and the terminal is analyzed according to the report.

[0164] The data processing method disclosed in the present disclosure has at least the following advantages:

[0165] On the one hand, the data is stored in a unified format, the collected data is in a unified format, the meaning is clear, and the management and task calling are convenient; the data development realizes batch calculation and stream calculation, and meets the needs of offline tasks and real-time calculation; the data assets gather the data of various business products, construct a data warehouse in layers, and construct a data mart in a subject domain, which can fully support various data services, realize data backfeeding to business, guide business, and optimize business.

[0166] On the other hand, since the data aggregation is realized by using a message engine, the purpose of traffic peak shaving can be achieved, and the data transmission channel can be realized. At the same time, since the data storage realizes the persistent storage of real-time data and offline data, the historical data can be calculated back or supplemented. In addition, the data development includes batch calculation and stream calculation, the batch calculation is suitable for high-latency processing scenarios of batch data, and the stream calculation is suitable for data processing and application with strong timeliness requirements.

[0167] On the other hand, the present disclosure realizes a data middle platform overall architecture construction method supporting software and hardware products. First, an interface method for collecting offline data sources or real-time data streams of software and hardware is provided. Then, the message engine is used to realize temporary storage of collected data. The data temporarily stored in the message engine is persistently stored, including data backup and data recovery functions. The data used by the data service is stored in a columnar database to ensure fast data query speed and support PB-level data volume. The data development can realize batch calculation and stream calculation, support batch processing after data accumulation, and support stream calculation of processing data as it comes, meet the needs of low-latency data processing. Based on the constructed data assets, various data services can be provided to realize data backfeeding to business, optimize business needs, and provide more empowerment to business.

[0168] Further, the present disclosure realizes obtaining source data on each hardware and software in an enterprise, and persistent storage, meeting the demand for historical data back calculation and data supplement; then obtaining enterprise business overall data assets through data development, forming a unified data management process; supporting business optimization and business innovation, the business optimization including improving efficiency, saving manpower, reducing cost and improving asset utilization rate; the business innovation such as digital product innovation, digital asset sales and digital ecological business. The digital product innovation is to realize incubation of new digital products by discovering new business value through data; the digital asset sales is to form data assets valuable for a part of users, such as industry reports or high-quality contents, by combining, packaging, analyzing and desensitizing the accumulated data, to directly sell and generate income; the digital ecological business is to help enterprises establish their own digital ecology by breaking through the industry supply chain, so as to generate new business value and income in the ecology.

[0169] The present disclosure also provides a data processing apparatus. Referring to Figure 11 The data processing apparatus can include a data extraction module 1110, a fact table forming module 1120, a fact table storage module 1130 and a target chart generation module 1140. Wherein:

[0170] The data extraction module 1110 can be used to extract raw data into a message engine based on a preset standardized data interface; wherein the raw data includes terminal data generated by a terminal device and / or system data generated by an application system;

[0171] The fact table forming module 1120 can be used to read the raw data from the message engine and store the raw data into a distributed file system to form a fact table;

[0172] The fact table storage module 1130 can be used to store the fact table into a columnar database and obtain target column data required for generating a target chart from the columnar database;

[0173] The target chart generation module 1140 can be used to generate the target chart according to the target column data.

[0174] In an exemplary embodiment of the present disclosure, based on a preset standardized data interface, raw data is extracted into a message engine, including:

[0175] Based on the preset standardized data interface, the terminal data pushed by a platform where the terminal device is located is received; wherein the terminal data includes one or more of device state behavior, command log data and hardware product data of the terminal device; and

[0176] Pull the system data from the relational database at intervals of a first preset time period based on a batch integration tool and a preset standardized data interface;

[0177] Converge the terminal data and / or the system data into the message engine.

[0178] In an exemplary embodiment of the present disclosure, converging the terminal data and / or the system data into the message engine comprises:

[0179] According to a data event type of the terminal data and / or the system data, converging the terminal data and / or the system data into the message engine under a topic corresponding to the data event type.

[0180] In an exemplary embodiment of the present disclosure, the fact table comprises a detailed fact table;

[0181] Wherein, reading the raw data from the message engine and storing the raw data into a distributed file system to form a fact table comprises:

[0182] Reading the raw data from the message engine and storing the raw data into the distributed file system under a date corresponding to a data occurrence timestamp of the raw data according to the data occurrence timestamp of the raw data;

[0183] Reading the raw data from the distributed file system and constructing a raw data layer according to the raw data;

[0184] According to the raw data included in the raw data layer and a data dictionary corresponding to the raw data, calculating a subject domain of the raw data;

[0185] According to the subject domain, summarizing the raw data to generate the detailed fact table.

[0186] In an exemplary embodiment of the present disclosure, reading the raw data from the distributed file system and constructing a raw data layer according to the raw data comprises:

[0187] Creating a raw data table comprising data source, data reporter, data version number, data sending location, data occurrence timestamp, data event type and data detailed information;

[0188] According to a storage location of the raw data in the distributed file system, reading the raw data;

[0189] According to a current date corresponding to a data occurrence timestamp of the raw data, writing the raw data into a position corresponding to the current date in the raw data table to obtain the raw data layer.

[0190] In an example embodiment of the present disclosure, a subject domain of the original data is calculated according to the original data included in the original data layer and a data dictionary corresponding to the original data, including:

[0191] A data dictionary corresponding to the original data is obtained, and a data service corresponding to the original data is determined according to the data dictionary;

[0192] A subject domain of the original data is determined according to the data service; wherein the subject domain includes one or more of a terminal domain, a system domain, and a user domain.

[0193] In an example embodiment of the present disclosure, a data dictionary corresponding to the original data is obtained, including:

[0194] When the original data is system data, metadata corresponding to the original data is pulled as the data dictionary;

[0195] When the original data is terminal data, the data dictionary is created according to a directory included in the metadata.

[0196] In an example embodiment of the present disclosure, the original data is summarized according to the subject domain to generate the detailed fact table, including:

[0197] When the original data is terminal data, the original data is summarized into a report with a subject domain of a terminal domain to generate a detailed fact table corresponding to the terminal domain;

[0198] When the original data is system data, the original data is summarized into a report with a subject domain of a system domain to generate a detailed fact table corresponding to the system domain.

[0199] In an example embodiment of the present disclosure, the fact table further includes a periodic snapshot fact table;

[0200] The data processing apparatus further includes:

[0201] The intermediate snapshot fact table generation module can be used to aggregate the detailed fact table based on a preset aggregation rule to obtain the periodic snapshot fact table;

[0202] The preset aggregation rule includes aggregation in a preset time interval.

[0203] In an example embodiment of the present disclosure, the detailed fact table is aggregated based on a preset aggregation rule to obtain the periodic snapshot fact table, including:

[0204] According to attribute information of the terminal device included in the terminal data and attribute information of the application system included in the system data, an attribute dimension table is generated;

[0205] According to common attributes between the terminal device and the application system included in the attribute dimension table, an association relationship between the detailed fact table corresponding to the terminal domain and the detailed fact table corresponding to the system domain is established;

[0206] Based on the association relationship, the detailed fact table corresponding to the terminal domain and the detailed fact table corresponding to the system domain are aggregated in a preset time interval, and the periodic snapshot fact table is obtained.

[0207] The specific details of each module in the above data processing apparatus have been described in detail in the corresponding data processing method, and therefore will not be described here.

[0208] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into several modules or units embodied.

[0209] In addition, although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, several steps can be combined into one step, and / or one step can be divided into several steps, etc.

[0210] In the exemplary embodiments of the present application, an electronic device capable of implementing the above method is also provided.

[0211] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method or a program product. Therefore, various aspects of the present application can be embodied as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" here.

[0212] The electronic device 1200 according to this embodiment of the present application will be described below with reference to Figure 12 The electronic device 1200 is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application. Figure 12 The electronic device 1200 is only an example and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0213] As Figure 12 shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 can include, but are not limited to, the at least one processing unit 1210 described above, the at least one storage unit 1220 described above, a bus 1230 that connects the different system components including the storage unit 1220 and the processing unit 1210, and a display unit 1240.

[0214] The storage unit stores program codes that can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present application described in the above "Exemplary Method" section of the present specification. For example, the processing unit 1210 can perform the steps shown in Figure 1 S110: extracting original data to a message engine based on a preset standardized data interface; wherein the original data includes terminal data generated by a terminal device and / or system data generated by an application system; S120: reading the original data from the message engine and storing the original data to a distributed file system to form a fact table; S130: storing the fact table to a columnar database and obtaining target column data required for generating a target chart from the columnar database; and S130: generating the target chart according to the target column data.

[0215] The storage unit 1220 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 12201 and / or a cache memory 12202, and can further include a read-only memory (ROM) 12203.

[0216] The storage unit 1220 can further include program / utility 12204 having a set of program modules 12205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof can include implementation of a network environment.

[0217] The bus 1230 can represent one or more of several types of bus structures, including a storage unit bus or storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit bus, or a local bus using any of a variety of bus architectures.

[0218] The electronic device 1200 can also communicate with one or more external devices 1300 such as a keyboard, a pointing device, a Bluetooth device, etc.; and can communicate with one or more devices that enable a user to interact with the electronic device 1200 and / or one or more devices that enable the electronic device 1200 to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 1250. Still yet, the electronic device 1200 can communicate with one or more networks such as a local area network (LAN), a wide area network (WAN), and / or the Internet through a network adapter 1260. As depicted, the network adapter 1260 communicates with the other components of the electronic device 1200 via the bus 1230. It should be appreciated that the electronic device 1200 can be a part of a larger system, and that there can be additional components that make up such system.

[0219] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by hardware coupled with software, as described above. Thus, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the methods according to the embodiments of the present application.

[0220] In the example embodiments of the present application, a computer readable storage medium is also provided, which stores a program product capable of implementing the above-mentioned methods of the present application. In some possible embodiments, various aspects of the present application can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps according to various example embodiments of the present application described in the above-mentioned “example method” section of the present specification when the program product is run on the terminal device.

[0221] The program product for implementing the above-mentioned methods according to the embodiments of the present application can be in the form of a portable compact disc read-only memory (CD-ROM) and includes program codes, and can be run on a terminal device such as a personal computer. However, the program product of the present application is not limited to this, and in the present document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0222] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0223] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0224] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0225] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0226] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0227] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

Claims

1. A data processing method, characterized in that, include: Based on a pre-defined standardized data interface, raw data is extracted into the message engine; wherein, the raw data includes terminal data generated by terminal devices and / or system data generated by application systems; The raw data is read from the message engine and stored in a distributed file system to form a fact table; wherein, the fact table includes a detailed fact table and / or a periodic snapshot fact table; The fact table is stored in a columnar database, and the target column data required to generate the target chart is obtained from the columnar database. The target chart is generated based on the target column data, and the operation status of the terminal devices and application systems included in the target chart is analyzed based on the target chart, including: analyzing / predicting the number of users using the terminal devices corresponding to the application system, and / or analyzing the number of times video data generated by the application system is played on the terminal devices, and / or analyzing the online status of the application system and the terminal devices. The detailed fact table is generated as follows: The raw data is read from the message engine, and stored in the distributed file system under the date corresponding to the data occurrence timestamp, based on the data occurrence timestamp of the raw data; the raw data is read from the distributed file system, and a raw data layer is constructed based on the raw data; the subject domain of the raw data is calculated based on the raw data included in the raw data layer and the data dictionary corresponding to the raw data; the raw data is summarized based on the subject domain to generate the detailed fact table; wherein the subject domain includes one or more of terminal domain, system domain, and user domain; The periodic snapshot fact table is generated as follows: An attribute dimension table is generated based on the attribute information of the terminal devices included in the terminal data and the attribute information of the application systems included in the system data; based on the common attributes between the terminal devices and the application systems included in the attribute dimension table, an association relationship is established between the detailed fact table corresponding to the terminal domain and the detailed fact table corresponding to the system domain; based on the association relationship, the detailed fact tables corresponding to the terminal domain and the detailed fact tables corresponding to the system domain are aggregated at preset time intervals to obtain the periodic snapshot fact table, which facilitates the analysis of the operational status of terminal devices and operating systems.

2. The data processing method according to claim 1, characterized in that, Based on a pre-defined standardized data interface, the raw data is extracted into the message engine, including: Based on a pre-defined standardized data interface, the system receives terminal data pushed by the platform where the terminal device resides; wherein, the terminal data includes one or more of the following: device status behavior, command log data, and hardware product data of the terminal device; and Based on batch integration tools and preset standardized data interfaces, the system data is retrieved from the relational database at first preset time intervals; The terminal data and / or the system data are aggregated into the message engine.

3. The data processing method according to claim 2, characterized in that, Aggregating the terminal data and / or the system data into the message engine includes: Based on the data event type of the terminal data and / or the system data, the terminal data and / or the system data are aggregated into the topic corresponding to the data event type in the message engine.

4. The data processing method according to claim 1, characterized in that, Reading the raw data from the distributed file system and constructing a raw data layer based on the raw data includes: Create a raw data table that includes data source, data reporter, data version number, data sending location, data occurrence timestamp, data event type, and detailed data information; The original data is read according to its storage location in the distributed file system; Based on the current date corresponding to the timestamp of the original data, the original data is written into the position corresponding to the current date in the original data table to obtain the original data layer.

5. The data processing method according to claim 4, characterized in that, Based on the raw data included in the raw data layer and the data dictionary corresponding to the raw data, the subject domain of the raw data is calculated, including: Obtain the data dictionary corresponding to the original data, and determine the data service corresponding to the original data based on the data dictionary; The subject domain of the original data is determined based on the data service.

6. The data processing method according to claim 5, characterized in that, Obtaining the data dictionary corresponding to the original data includes: When the original data is system data, the metadata corresponding to the original data is retrieved as the data dictionary; When the raw data is terminal data, the data dictionary is created based on the directory included in the metadata.

7. The data processing method according to claim 6, characterized in that, The raw data is summarized based on the subject area to generate the detailed fact table, including: When the original data is terminal data, the original data is summarized into a report whose subject area is the terminal area, and a detailed fact table corresponding to the terminal area is generated. When the original data is system data, the original data is summarized into a report with the subject domain as the system domain, and a detailed fact table corresponding to the system domain is generated.

8. A data processing platform, comprising: The data aggregation layer extracts raw data into the message engine based on a pre-defined standardized data interface; A data storage layer, connected to the data aggregation layer, is used to read the raw data from the message engine and store the raw data in a distributed file system; A data processing layer, connected to the data storage layer, is used to read the raw data from the distributed file system and generate a fact table based on the raw data; wherein the fact table includes a detailed fact table and / or a periodic snapshot fact table; and The fact table is stored in a columnar database, and the target column data required to generate the target chart is retrieved from the columnar database; and The target chart is generated based on the target column data, and the operation status of the terminal devices and application systems included in the target chart is analyzed based on the target chart, including: analyzing / predicting the number of users using the terminal devices corresponding to the application system, and / or analyzing the number of times video data generated by the application system is played on the terminal devices, and / or analyzing the online status of the application system and the terminal devices. The detailed fact table is generated as follows: The raw data is read from the message engine, and stored in the distributed file system under the date corresponding to the data occurrence timestamp, based on the data occurrence timestamp of the raw data; the raw data is read from the distributed file system, and a raw data layer is constructed based on the raw data; the subject domain of the raw data is calculated based on the raw data included in the raw data layer and the data dictionary corresponding to the raw data; the raw data is summarized based on the subject domain to generate the detailed fact table; wherein the subject domain includes one or more of terminal domain, system domain, and user domain; The periodic snapshot fact table is generated as follows: An attribute dimension table is generated based on the attribute information of the terminal devices included in the terminal data and the attribute information of the application systems included in the system data; based on the common attributes between the terminal devices and the application systems included in the attribute dimension table, an association relationship is established between the detailed fact table corresponding to the terminal domain and the detailed fact table corresponding to the system domain; based on the association relationship, the detailed fact tables corresponding to the terminal domain and the detailed fact tables corresponding to the system domain are aggregated at preset time intervals to obtain the periodic snapshot fact table, which facilitates the analysis of the operational status of terminal devices and operating systems.

9. The data processing platform according to claim 8, characterized in that, The data processing layer includes a raw data layer, a unified data warehouse layer, an application data layer, a tag data layer, and dimension tables; The unified data warehouse layer is connected to the original data layer, the application data layer and the tag data layer are respectively connected to the unified data warehouse layer, and the dimension table is contained in the unified data warehouse layer; The raw data layer is used to read the raw data from the file storage system and store the raw data. The unified data warehouse layer is used to calculate the subject domain of the original data based on the original data included in the original data layer and the data dictionary corresponding to the original data; and The raw data is summarized based on the subject area to generate a detailed fact table; and Based on the attribute information of terminal devices in the terminal data included in the original data, and the attribute information of application systems in the system data included in the original data, an attribute dimension table is generated; and Based on the detailed fact table and the attribute dimension table, generate a periodic snapshot fact table; The application data layer is used to obtain the target column data required to generate the target chart from the columnar database, and generate the target chart based on the target column data; The tag data layer is used to identify the terminal devices and application systems included in the raw data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1-7.

11. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1-7 by executing the executable instructions.

Images