A data processing system and a data processing method

Abstract

The present application provides a data processing system and a data processing method, which are applied to the field of data processing technology. The system includes: a data synchronization module and a data warehouse. The data synchronization module is used to obtain business data from multiple data sources, and based on the synchronization method corresponding to the data source, the business data is synchronously stored in the data warehouse. The data source includes: irregular data and regular data. Regular data includes one or more of the following: service provider system, database, and local files generated by the enterprise. The data warehouse includes: ODS layer, DW layer, and ADS layer; the data warehouse is used to input business data into the ODS layer; the DW layer is used to configure the business data to obtain wide table data; the wide table data is used to represent business data of multiple dimensions under a preset theme; the ADS layer is used to call the business data corresponding to the user's needs from the wide table data according to the user's needs, and generate a business data table, which can realize data interoperability and improve data management efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing system and a data processing method. Background Art

[0002] With the rapid development of information technology, traditional manufacturing enterprises are gradually transforming to intelligent manufacturing, which generates a huge amount of business data. However, these business data are often stored in different product service providers for corresponding data management, and the distribution is relatively scattered. At the same time, due to the lack of intercommunication between different product service providers, these business data form data islands, making it difficult for enterprises to effectively integrate and utilize these business data, ignoring the hidden value in the business data.

[0003] Based on this, the present application provides a data processing system and a data processing method for achieving data interoperability and improving data management efficiency. Summary of the invention

[0004] The purpose of this application is to provide a data processing system and a data processing method to achieve data interoperability and improve data management efficiency.

[0005] In the first aspect, an embodiment of the present application provides a data processing system, which includes: a data synchronization module and a data warehouse. The data synchronization module is used to obtain business data from multiple data sources, and based on the synchronization method corresponding to the data source, synchronize and store the business data to the data warehouse. The data source includes: irregular data and regular data. The regular data includes one or more of the following: service provider system, database, and local files generated by the enterprise. The data warehouse includes: ODS layer, DW layer, and ADS layer; the data warehouse is used to input the business data into the ODS layer; the business data is configured using the DW layer to obtain wide table data; the wide table data is used to represent the business data of multiple dimensions under a preset theme; the ADS layer is used to call the business data corresponding to the user needs from the wide table data according to user needs to generate the business data table.

[0006] The data processing system provided in the embodiment of the present application can solve the problems of scattered data and the inability to centrally manage in the past. By deploying and building a data processing system on a cloud service, the pressure on the operation and maintenance of the previously deployed proprietary clusters is reduced, the reliability of the service is guaranteed, and various forms of data can be managed. For different business needs, it is also more convenient to add and reduce the required functions, and the scalability is good. At the same time, by using the data quality monitoring, data overview and other services provided by the cloud service, the data in the data center can be better monitored, the accuracy of the development data can be guaranteed, and work efficiency can be improved.

[0007] In a possible implementation, the data content of the business data is irregular data, and the data synchronization module is specifically used to obtain the data type of the business data, and based on the data type, store the business data in the data warehouse. The data type includes one or more of the following: data with a data volume less than a preset data volume, record table data recording detailed action information, synchronized data with update changes, data stored in file form, and data with deletion behavior.

[0008] In a possible implementation, the data source is a service provider system, and the data synchronization module is specifically used to obtain the business data from the service provider system according to a preset time through the SDK service provided by the service provider system, and schedule the business data to the data warehouse.

[0009] In a possible implementation, the data source is a database, and the data synchronization module is specifically used to obtain the business data in the database in a JDBC manner through a database interface provided by the database, and synchronously store the business data in the data warehouse.

[0010] In a possible implementation, the data source is a local file generated by an enterprise, and the data synchronization module is specifically used to synchronize the business data increments to the data warehouse when a change in the business data is detected.

[0011] In a possible implementation, the data warehouse includes: an ODS layer, a DW layer, and an ADS layer. The data warehouse is specifically used to input the business data into the ODS layer. The DW layer is used to configure the business data to obtain wide table data. The wide table data is used to represent the business data of multiple dimensions under a preset theme. The ADS layer is used to call the business data corresponding to the user's needs from the wide table data according to user needs to generate the business data table.

[0012] In a possible implementation, the DW layer includes: a DWP layer, a DWD layer, and a DWS layer. The data warehouse is specifically used to use the DWP layer to partition and store the business data when the amount of the business data is greater than a threshold. The DWD layer is used to divide the business data into fact tables of multiple dimensions according to the preset subject and the data type of the business data. The DWS layer is used to summarize the fact tables to obtain the wide table data. The wide table data corresponds to the preset subject.

[0013] In a possible implementation, the system further includes: a data monitoring module. The data monitoring module is used to obtain real-time tasks of the data synchronization module and the data warehouse. The real-time tasks include the data synchronization module acquiring business data from multiple data sources, the data synchronization module synchronously storing the business data in the data warehouse based on a synchronization method corresponding to the data source, and the data warehouse generating a business data table based on the data type of the business data. The real-time tasks are scheduled based on the task information of the real-time tasks. The task information includes the occupied resources, duration and priority information of the real-time tasks.

[0014] In a possible implementation, the system further includes: a data quality management module. The data quality management module is used to monitor the real-time data volume information in the data warehouse in real time. The real-time data volume information is used to represent the data volume information of the business data table. The real-time data volume information is compared with the historical data volume information of the data warehouse to determine the data quality change of the data warehouse.

[0015] In a possible implementation, the system further includes: a data application module. The data application module is used to process and export the business data table according to a preset data export method. The preset data export method includes visual display of the business data table based on QuickBI, sending the business data table to a third-party platform based on an interface service, and performing NLP processing on the business data table.

[0016] In a second aspect, an embodiment of the present application provides a data processing method, the method comprising: obtaining business data from multiple data sources using a data synchronization module, and synchronously storing the business data in the data warehouse based on a synchronization method corresponding to the data source. The data source comprises: irregular data and regular data. The regular data comprises one or more of the following: a service provider system, a database, and a local file generated by an enterprise. The data warehouse is used to generate a business data table based on the data type of the business data.

[0017] In a third aspect, the present application provides a data processing device having the function of implementing the data processing method of the first aspect or any possible implementation method. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0018] In a fourth aspect, a computer-readable storage medium is provided, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer can execute the data processing method of the first aspect or any possible implementation method.

[0019] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the data processing method of the first aspect or any possible implementation manner.

[0020] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different possible implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A structural schematic diagram of a data processing system provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the structure of a data warehouse provided in an embodiment of the present application;

[0024] Figure 3 A specific example diagram of a data processing system provided in an embodiment of the present application;

[0025] Figure 4 Another specific example diagram of a data processing system provided in an embodiment of the present application;

[0026] Figure 5 Another structural schematic diagram of a data processing system provided in an embodiment of the present application;

[0027] Figure 6 A method flow chart of a data processing method provided in an embodiment of the present application;

[0028] Figure 7 Another structural schematic diagram of a data processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0030] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0031] First, the terms of the embodiments of the present application are explained.

[0032] Mining is to extract data stored in OSS / file system through programs such as Python / Java, transfer the original data and extracted data to the data warehouse, and classify the data using relevant data mining methods to provide a finer statistical granularity for subsequent data processing.

[0033] Intercommunication is to utilize the huge throughput of OSS / Kafka, which can directly transform and process data, and can transmit and respond to data faster and more conveniently. For example, by utilizing OSS's storage capacity for unstructured data, we will process, compress, and encrypt the data collected from Kafka through Flink, package it into a zip package, and provide data services to the outside.

[0034] At present, the data of the traditional manufacturing industry is managed by product service providers, and the distribution is relatively scattered. At the same time, due to the lack of intercommunication between different product service providers, these business data have formed data islands, making it difficult for enterprises to effectively integrate and utilize these business data, ignoring the hidden value in business data.

[0035] Based on this, an embodiment of the present application provides a data processing system and a data processing method. The data processing system includes: a data synchronization module and a data warehouse.

[0036] Among them, the data synchronization module is used to obtain business data from multiple data sources, and based on the synchronization method corresponding to the data source, synchronize and store the business data to the data warehouse. Data sources include: irregular data and regular data. Regular data includes one or more of the following: service provider systems, databases, and local files generated by enterprises. The data warehouse includes: ODS layer, DW layer, and ADS layer; the data warehouse is used to input business data into the ODS layer; the DW layer is used to configure business data to obtain wide table data; wide table data is used to represent business data of multiple dimensions under preset themes; the ADS layer is used to call business data corresponding to user needs from wide table data according to user needs to generate business data tables, which can realize data interoperability and improve data management efficiency.

[0037] On the one hand, the embodiment of the present application provides a data processing system 100, which can be in the form of a data middle station, and the present application is not limited to this. Figure 1 As shown, the data processing system may include: a data synchronization module 101 and a data warehouse 102 .

[0038] The data synchronization module 101 is specifically used to obtain business data from multiple data sources, and synchronously store the business data in a data warehouse based on a synchronization method corresponding to the data source.

[0039] The data content of the business data may include regular data and irregular data.

[0040] Data sources include one or more of the following: service provider systems, databases, local files generated by the enterprise, and object storage systems.

[0041] In a possible implementation, the process of the data synchronization module 101 acquiring business data from the data source may include: when the data source is a service provider system, the data synchronization module 101 may acquire business data that provides a relevant business API interface from the service provider system. The data synchronization module 101 may also acquire business data from other cloud service provider systems, such as Alibaba Cloud's OSS, MaxCompute service, etc.

[0042] The data synchronization module 101 obtains business data from the service provider system according to a preset time through the SDK service provided by the service provider system, and schedules the business data to the data warehouse.

[0043] Exemplarily, when the data source provides an API interface and is stored in other cloud service providers, by writing a program package based on Java or Python, etc. in the data synchronization module 101, and deploying the data synchronization module 101 in the server on the service, scheduling is performed regularly (for example, synchronization is performed once a day, or the synchronization time interval can be determined according to specific business needs), and the business data is uploaded to the data warehouse.

[0044] Another possible implementation is that when the data source is a database, the data synchronization module 101 can obtain the business data generated by the enterprise backend from the database. The business data is usually stored in a database deployed by the enterprise server. For example, the database can be: MySQL, Oracle DB, MongoDB, etc.

[0045] The data synchronization module 101 is specifically used to obtain the business data in the database in a JDBC manner through the database interface provided by the database, and synchronize and store the business data in the data warehouse.

[0046] Exemplarily, when the data source is a database, the data synchronization module 101 accesses the database in JDBC mode through the data source interface provided by the cloud service DataWork (a cloud service provided by Alibaba Cloud that integrates data storage, development, and management), and obtains business data from the database. It is also assisted by some open source software for data synchronization, such as datax (connecting to the database through JDBC, using SQL statements to query data in the database, and transmitting data to a designated downstream server). The open source software for data synchronization can also be Sqoop, and flinkCDC based on database log data synchronization, etc.

[0047] When the data source is a local file generated by an enterprise, the data synchronization module 101 can obtain a log file generated by a local system from the local file generated by the enterprise. The log file can be stored in local files of servers of different service providers.

[0048] The data synchronization module 101 is specifically used to synchronize the business data increments to the data warehouse when a change in the business data is detected.

[0049] Exemplarily, when the data source is a local file generated by an enterprise, the data synchronization module 101 can use the data collection framework Flume to monitor file changes and send data to the downstream for log collection, and send the collected data to the distributed message queue framework (Kafka) or directly send the data to Kafka during the service operation for downstream processing.

[0050] The process of the data synchronization module 101 acquiring business data from the data source may be that the data synchronization module 101 acquires data from the object storage system OSS. The data may be data such as documents, pictures, and web pages.

[0051] The data synchronization module 101 is specifically used to obtain the data type of the business data, and based on the data type, store the business data in the data warehouse. The data type includes one or more of the following: data with a data volume less than a preset data volume, record table data recording detailed action information, synchronized data with updated changes, data stored in the form of files, and data with deletion behavior.

[0052] Exemplarily, for business data in the data source whose data type is business data with a business data volume less than a preset data volume, the data synchronization module 101 directly synchronizes the full amount of business data to the data warehouse.

[0053] For business data in the data source whose data type is record table data for recording detailed action information, the data synchronization module 101 synchronizes the business data to the data warehouse according to the date field of the business data. If the business data does not have a date field, the business data can be recorded with a synchronized auto-increment id. During the data synchronization process, the largest id in the synchronized data is first found using the auto-increment id, and then the data with the data id greater than the synchronized largest id is filtered out from the data source for synchronization.

[0054] When synchronizing data with updated data in the data source, the data synchronization module 101 selects data with both updated date and created date on the same day and synchronizes them to the data warehouse. The data can be pre-processed through the DW layer, and the latest data can be obtained in chronological order to ensure that the data is consistent with the data source, and the data change information is recorded at the same time.

[0055] For business data in the data source that is stored in a file or other form, the data synchronization module 101 first compresses the business data and then transmits the business data to the data warehouse through the network, which can reduce network costs and pressure.

[0056] For business data with deletion behavior in the data source, the data synchronization module 101 can add a logical deletion field to the business data at the business level. When the business data cannot add a logical deletion field, the data synchronization module 101 can read the database binlog log mode, synchronize the primary key id field, and then the data warehouse processes the business data and restores it to the dwd layer.

[0057] The data warehouse 102 is used to generate a business data table based on the data type of the business data.

[0058] Among them, Figure 1 As shown, the data warehouse 102 may include: an ODS layer, a DW layer, and an ADS layer.

[0059] In one possible implementation, the data warehouse is specifically used to input business data into the ODS layer. The DW layer is used to configure the business data to obtain wide table data. The wide table data is used to represent business data of multiple dimensions under a preset theme. The ADS layer is used to call business data corresponding to user needs from the wide table data according to user needs to generate a business data table.

[0060] Exemplarily, the ODS layer is the first layer where business data is stored in the data warehouse, and the ODS layer stores the original data synchronized by the data synchronization module.

[0061] The DW layer is used to configure business data and obtain business data of multiple dimensions under preset topics.

[0062] The ADS layer is used to obtain business data of multiple dimensions under preset topics from the DW layer, and call business data corresponding to user needs from the wide table data according to user needs to generate a business data table.

[0063] A possible implementation is Figure 2 As shown, the DW layer of the data warehouse 102 may specifically include: a DWP layer, a DWD layer, and a DWS layer.

[0064] The data warehouse is specifically used to use the DWP layer to partition and store business data when the amount of business data is greater than a threshold. The DWD layer is used to divide business data into fact tables of multiple dimensions according to preset topics and data types of business data. The DWS layer is used to summarize the fact tables to obtain wide table data. The wide table data corresponds to the preset topics.

[0065] Exemplarily, the DWP layer is used to pre-process the business data in the ODS layer. If the amount of business data is less than a threshold, the business data is stored as a table. If the amount of business data is greater than a threshold, the business data is partitioned. For example, for data with updates and changes, the business data is partitioned by creation date and update date in the DWP layer, and files with larger data volumes are processed as updated by creation date. This can improve the efficiency of downstream data processing.

[0066] The DWD layer is used to divide business data into topics and build fact tables at different granularities. For example, topics can be divided into three modules: users, products, and vehicles. At the same time, fact tables of different granularities are divided based on the dynamically changing data and static data of the topic, and one id is reassigned. This ensures that the original data between various systems will not conflict when supporting the upper-layer data, and at the same time ensures the uniqueness of the data for the final data output.

[0067] The DWS layer is used to lightly summarize the DWD layer data to obtain wide table data that can be used for public use to target different business modules.

[0068] ADS is specifically used to develop specific tables according to specific business needs after receiving the wide table data of the DW layer, and output them to the required business, such as BI reports, specific data required by users, and API interface data. This level directly outputs data to the outside world, and should be as specific as possible to reduce the association between data, so as to improve the efficiency of calling for data demanders.

[0069] A possible implementation is Figure 2 As shown, the data warehouse 102 may also include: a DIM layer and a TMP layer.

[0070] The DIM layer is used to store common dimension data. Based on the concept of dimensional modeling, a consistent dimension can be established, which is a mapping relationship table of id-other_id. At this level, some tables are stored in hbase for real-time computing, and some are stored in MaxComputer.

[0071] The TMP layer is used to store temporary data generated by other layers in the data warehouse during the calculation process.

[0072] A possible implementation method for data warehouse development may include: using MaxComputer's odps sql to write SQL and process the flow of data between layers. Alternatively, use the open source computing engine Spark for offline data development. Alternatively, use the open source computing engine flink to process real-time App application data obtained from kafka, system log monitoring data and other data with high real-time requirements, use the distributed nosql database (hbase) to store the dimension table in the DIM layer, and after calculation, use kafka to store the data and send it to the next layer, and finally store the data in the online analytical processing query framework (clickhouse).

[0073] For example, Figure 3 As shown in the figure, taking user behavior, log and user attribute data as an example, based on the original system, the main behavior data processing flow of the user is to first upload the user's specific behavior results, log data, etc. from the data uploaded to the server to Kafka, and then perform real-time processing through Flink, combined with Redis, HBase and different API interfaces to achieve data accumulation, filtering, interaction and supplement. In this process, due to Kafka's high availability, high throughput, cloud native and disk storage mechanism, there is no need to consider the maintainability of components too much. In the event of an accident, the data can be restored, and the data security can be guaranteed as much as possible.

[0074] Then, through the Kafka-into-data-warehouse function included in MaxComputer, the data is stored in the ODS layer of the warehouse. At the same time, a relatively high compression algorithm is used to save space and archive and back up the data.

[0075] Once again, through Maxcomputer's Kafka warehousing function, the data processed by Flink, which is more complex and high-dimensional than the original data, is directly imported into the DWP layer. For changing business needs, data can also be merged into the data warehouse by expanding different topics, reducing development complexity and improving development efficiency.

[0076] Furthermore, different types of processing services are used to perform different processing, such as directly connecting with customers to display the data they need, reading data from the data warehouse for statistics and secondary development, etc.; then the data is cleaned, standardized, and pre-aggregated and stored in the DWD layer, and a scheduled task is started to merge small files and separate hot and cold data on the DWP layer data to reduce data storage space and improve data analysis efficiency. In the DWD layer, the data is processed in the corresponding business logic direction, such as combining user orders, user attribute data, etc., to develop corresponding user portrait attributes for users, analyze possible problems or vulnerabilities of products through user behavior, and perform trend analysis through data statistics, and produce corresponding data of different dimensions, which are stored in the DWS layer table. Finally, data of different dimensions are aggregated, classified, and counted, and the data is output to the ADS layer. The output data can be exported through datax / API / SDK / BI / clickhouse / starrocks, etc., or provide basic data for artificial intelligence methods such as neural networks, to implement question-answering systems, product price estimation, etc., to achieve higher-level data value mining and analysis. For data with high real-time requirements, you can use vectorized OLTP computing engines such as clickhouse and starrock or Redis and HBase based on kv storage to achieve high real-time performance. When real-time performance is not a must, you can use Maxcomputer's query acceleration service, Datax, etc. to achieve quasi-real-time and offline business.

[0077] For example, Figure 4 As shown in the figure, for unstructured data processing, such as HTML web pages, PDF, pictures and other data. First, store the data in OSS, use scripts to perform cluster analysis on the data, such as decision trees and other methods. For foreseeable collection types such as HTML pages, first parse the HTML, and then extract the tags and attributes of the HTML pages in turn. By controlling the threshold range, repeatedly recursively extract the tag names and attributes of multiple same-level and sub-tags, and store the data in the ODS level of the data warehouse in the form of fields.

[0078] The data in the ODS layer is standardized and cleaned, such as deduplication and null removal or corresponding business processing to store the DWD layer data; then the DWD hierarchical table is analyzed by clustering algorithm and calibrated with some human intervention, and the ordered tags and attribute fields of the extracted web page HTML are marked with data classification labels, and the resulting data is saved as a DIM-level dimension table. The DWD layer data is used to associate the above-produced DIM dimension table data with the parsed ordered HTML tags and attributes to obtain the DWS layer data, and the classified page data is obtained and standardized, and specific business logic processing of different categories is performed, etc. Finally, according to specific business needs, the data is aggregated, counted, extracted, and other operations are performed, and the data is output to the ADS layer or exported to other external components such as clickhouse and OSS for external use.

[0079] like Figure 5 As shown, the data processing system may further include: a data monitoring module 103 , a data quality management module 104 and a data application module 105 .

[0080] Among them, the data monitoring module 103 is used to obtain real-time tasks of the data synchronization module and the data warehouse. The real-time task includes the data synchronization module obtaining business data from multiple data sources, the data synchronization module synchronously storing the business data in the data warehouse based on the synchronization method corresponding to the data source, and the data warehouse generating a business data table based on the data type of the business data. The real-time task is scheduled based on the task information of the real-time task. The task information includes the occupied resources, duration and priority information of the real-time task.

[0081] Exemplarily, the data monitoring module 103 can rely on MaxComputer and its external interface services to monitor currently running tasks in real time. Based on the current cluster available resources and all resources, analyze the current tasks that occupy too many resources and take too long, and send messages to the R&D group or the corresponding personnel mailbox to promptly remind developers to optimize tasks. The data monitoring module 103 can also increase the priority of key tasks such as daily scheduling tasks and external service tasks based on historical task scheduling and duration to ensure the normal operation of key tasks and the normal external service of the cluster. The data monitoring module 103 can also analyze the space occupied by data, file modification time, tables that have not been used for a long time, the amount of data in table partitions, and other information through data metadata in the database, and prompt developers to separate and compress cold data in time, optimize the table structure to improve the efficiency of subsequent task operations, delete useless tables to save disk space, etc.

[0082] The data quality management module 104 is used to monitor the real-time data volume information in the data warehouse in real time. The real-time data volume information is used to represent the data volume information of the business data table. The real-time data volume information is compared with the historical data volume information of the data warehouse to determine the data quality change of the data warehouse.

[0083] For example, the data quality management module 104 can monitor business data, maintain the normal operation of business data tasks, and send email prompts for error tasks through the services provided by MaxComputer. Data quality is controlled through data auditing, and attention is paid to empty data, problematic data, etc. The data overview and data dependency display the blood relationship between data, which can improve the developer's understanding of the data and better sort out the relationship between business data.

[0084] The data quality management module 104 can also be used to build a data quality management system, monitor the data volume changes of pre-marked key tables in the data warehouse, and obtain information such as whether the corresponding table has failed to synchronize data in a timely manner, poor data quality, abnormal data fluctuations, etc. by retaining the historical data volume of the table, analyzing data changes, and counting information such as the proportion of null values. The statistical information is sent to the relevant personnel on a daily basis to ensure that developers can understand data quality changes in a timely manner, and at the same time ensure that the data can be correctly provided to external services.

[0085] The data application module 105 is used to process and export the business data table according to a preset data export method. The preset data export method includes visual display of the business data table based on QuickBI, sending the business data table to a third-party platform based on an interface service, and performing NLP processing on the business data table.

[0086] Exemplarily, the data application module 105 is the data export of the data processing system, and can provide business demand data to the outside by developing SDKs, APIs, etc., discover possible problems in enterprise data, explore the value of data, and provide feedback for business operations and decision-making.

[0087] For example, through QuickBI, corresponding business data statistics are displayed, such as daily active users, user consumption preferences, product usage, etc., and the data is visualized in the form of statistical charts such as line charts, pie charts, word clouds, etc., to understand user device usage and user activity information. Through statistical images, combined with specific time and related specific data events, it is possible to determine whether the product has bugs, provide data support and reference suggestions for improvement directions for the development of other team members, thereby improving user experience.

[0088] By developing API interfaces, we provide data to partners, accurately analyze user behavior, infer potential value, push relevant updates to users, iterate notifications, and statistically analyze reach and positive feedback ratios to form a user-user closed loop; directly provide users with basic data service support.

[0089] Through artificial intelligence, machine learning and other NLP processing, we can determine the main meaning of the user's text data and understand the issues that the user is most concerned about. From the generated data, we can infer the category to which the user's new data question may belong, and use the trained model to give possible answers to the question, so as to provide users and decision makers with better solutions.

[0090] The data processing system provided in the embodiment of the present application can solve the problems of scattered data and the inability to centrally manage in the past. By deploying and building a data processing system on a cloud service, the pressure on the operation and maintenance of the previously deployed proprietary clusters is reduced, the reliability of the service is guaranteed, and various forms of data can be managed. For different business needs, it is also more convenient to add and reduce the required functions, and the scalability is good. At the same time, by using the data quality monitoring, data overview and other services provided by the cloud service, the data in the data center can be better monitored, the accuracy of the development data can be guaranteed, and work efficiency can be improved.

[0091] On the one hand, the present application embodiment provides a data processing method, such as Figure 6 As shown, the method includes:

[0092] S601, using a data synchronization module to obtain business data from multiple data sources, and based on a synchronization method corresponding to the data source, synchronously storing the business data in a data warehouse.

[0093] The data content of business data includes: irregular data and regular data. The data source includes one or more of the following: service provider system, database, and local files generated by the enterprise.

[0094] It should be noted that the process of using the data synchronization module to obtain business data from multiple data sources and synchronously storing the business data in the data warehouse based on the synchronization method corresponding to the data source can refer to the function of the data synchronization module in the above-mentioned data processing system, and this application will not go into details.

[0095] S602, using the data warehouse to input the business data into the ODS layer of the data warehouse, using the DW layer of the data warehouse to configure the business data to obtain wide table data, using the ADS layer of the data warehouse to call the business data corresponding to the user needs from the wide table data according to user needs, and generate a business data table.

[0096] Among them, wide table data is used to represent business data of multiple dimensions under a preset theme.

[0097] It should be noted that the process of using the data warehouse to input business data into the ODS layer of the data warehouse; using the DW layer of the data warehouse to configure the business data to obtain wide table data; the wide table data is used to represent business data of multiple dimensions under a preset theme; using the ADS layer of the data warehouse to call business data corresponding to user needs from the wide table data according to user needs to generate a business data table can refer to the function of the data warehouse in the above-mentioned data processing system, and this application is not limited to this.

[0098] The embodiment of the present application also provides a data processing device, which can be used for deploying the above-mentioned Figure 1 The data processing system shown in the figure can execute the data processing method and related steps in the above method embodiment.

[0099] An embodiment of the present application also provides a computer-readable storage medium on which instructions are stored. When the instructions are executed, the data processing method and related steps in the above method embodiment are executed.

[0100] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes the data processing method and related steps in the above method embodiment.

[0101] In some embodiments, the methods described herein may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or encoded on other non-transitory media or articles of manufacture.

[0102] The present application embodiment also provides a data processing system 700, such as Figure 7 As shown, the data processing system 700 includes at least one processor 701 and at least one interface circuit 702 .

[0103] As an example, when the data processing system 700 includes a processor and an interface circuit, the processor may be Figure 7 The processor 701 shown in the solid line frame (or the processor 701 shown in the dotted line frame) may be Figure 7 The interface circuit 702 shown in the solid line frame (or the interface circuit 702 shown in the dotted line frame). When the chip system 700 includes two processors and two interface circuits, the two processors include Figure 7 The processor 701 shown in the solid line frame and the processor 701 shown in the dotted line frame, the two interface circuits include Figure 7 The interface circuit 702 shown in the solid line frame and the interface circuit 702 shown in the dotted line frame are not limited to this.

[0104] The processor 701 and the interface circuit 702 may be interconnected via a line. For example, the interface circuit 702 may be used to receive a signal. In another example, the interface circuit 702 may be used to send a signal to another device (e.g., the processor 701). For example, the interface circuit 702 may read a computer instruction stored in a memory and send the computer instruction to the processor 701. The processor 701 executes the instruction and implements the various steps in the above embodiments in combination with the input and output devices, such as implementing Figure 6 The various steps performed in the method embodiment shown. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0105] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0106] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0110] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing system, characterized in that: The system includes: a data synchronization module and a data warehouse; The data synchronization module is used to obtain business data from multiple data sources, and based on the synchronization method corresponding to the data source, synchronize and store the business data to the data warehouse; the data content of the business data includes: irregular data and regular data; the data source includes one or more of the following: service provider system, database, and local files generated by the enterprise; The data warehouse includes: an ODS layer, a DW layer, and an ADS layer; the data warehouse is used to input the business data into the ODS layer; the DW layer is used to configure the business data to obtain wide table data; the wide table data is used to represent the business data of multiple dimensions under a preset theme; the ADS layer is used to call the business data corresponding to the user's needs from the wide table data according to the user's needs to generate a business data table; The data content of the business data is irregular data, and the data synchronization module is specifically used to obtain the data type of the business data, and based on the data type, store the business data in the data warehouse; the data type includes one or more of the following: data with a data volume less than a preset data volume, record table data recording detailed action information, synchronized data with update changes, data stored in the form of files, and data with deletion behavior; For business data in the data source whose data type is record table data that records detailed action information, the data synchronization module synchronizes the business data to the data warehouse according to the date field of the business data; if the business data does not have a date field, the business data records the synchronized auto-increment id; during the data synchronization process, the auto-increment id is used to first find the largest id in the synchronized data, and then the data with id greater than the synchronized largest id is filtered out from the data source for synchronization; When synchronizing data with updated data in the data source, the data synchronization module selects data with both updated date and created date on the same day and synchronizes them to the data warehouse; pre-processes the data through the DW layer, takes the latest data in chronological order, ensures that the data is consistent with the data source, and records the data change information; For business data in the data source that is business data stored in files, the data synchronization module first compresses the business data and then transmits the business data to the data warehouse through the network; For business data with deletion behavior in the data source, the data synchronization module adds a logical deletion field to the business data at the business level; when the business data cannot add a logical deletion field, the data synchronization module reads the database binlog log, synchronizes the primary key ID field, and then the data warehouse processes the business data and restores it to the DWD layer.

2. The system according to claim 1, characterized in that The data source is a service provider system, and the data synchronization module is specifically used to obtain the business data from the service provider system according to a preset time through the SDK service provided by the service provider system, and schedule the business data to the data warehouse.

3. The system according to claim 1, characterized in that The data source is a database, and the data synchronization module is specifically used to obtain the business data in the database in a JDBC manner through a database interface provided by the database, and synchronously store the business data in the data warehouse.

4. The system according to claim 1, characterized in that The data source is a local file generated by the enterprise, and the data synchronization module is specifically used to synchronize the business data increments to the data warehouse when a change in the business data is detected.

5. The system according to claim 1, characterized in that The DW layer includes: a DWP layer, a DWD layer, and a DWS layer; The data warehouse is specifically used to use the DWP layer to partition and store the business data when the data volume of the business data is greater than a threshold; use the DWD layer to divide the business data into fact tables of multiple dimensions according to the preset subject and the data type of the business data; use the DWS layer to summarize the fact tables to obtain the wide table data; the wide table data corresponds to the preset subject.

6. The system according to claim 1, characterized in that The system also includes: a data monitoring module; The data monitoring module is used to obtain real-time tasks of the data synchronization module and the data warehouse; the real-time tasks include the data synchronization module acquiring business data from multiple data sources, the data synchronization module synchronously storing the business data to the data warehouse based on the synchronization method corresponding to the data source, and the data warehouse generating a business data table based on the data type of the business data; scheduling the real-time tasks based on the task information of the real-time tasks; the task information includes the occupied resources, duration and priority information of the real-time tasks.

7. The system according to claim 1, characterized in that The system also includes: a data quality management module; The data quality management module is used to monitor the real-time data volume information in the data warehouse in real time; the real-time data volume information is used to represent the data volume information of the business data table; the real-time data volume information is compared with the historical data volume information of the data warehouse to determine the data quality changes of the data warehouse.

8. The system according to claim 1, characterized in that The system further comprises: a data application module; The data application module is used to process and export the business data table according to a preset data export method; the preset data export method includes visual display of the business data table based on QuickBI, sending the business data table to a third-party platform based on an interface service, and performing NLP processing on the business data table.

9. A data processing method, characterized in that: The method comprises: The data synchronization module is used to obtain business data from multiple data sources, and based on the synchronization method corresponding to the data source, the business data is synchronously stored in the data warehouse; the data content of the business data includes: irregular data and regular data; the regular data includes one or more of the following: service provider system, database, and local files generated by the enterprise; The business data is input into the ODS layer of the data warehouse by using the data warehouse; the business data is configured by using the DW layer of the data warehouse to obtain wide table data; the wide table data is used to represent the business data of multiple dimensions under a preset theme; the business data corresponding to the user demand is called from the wide table data by using the ADS layer of the data warehouse according to the user demand to generate a business data table; The data content of the business data is irregular data, the data type of the business data is obtained by using the data synchronization module, and based on the data type, the business data is stored in the data warehouse; the data type includes one or more of the following: data with a data volume less than a preset data volume, record table data recording detailed action information, synchronized data with update changes, data stored in the form of files, and data with deletion behavior; For business data in the data source whose data type is record table data that records detailed action information, the data synchronization module is used to synchronize the business data to the data warehouse according to the date field of the business data; if the business data does not have a date field, the synchronized auto-increment id is recorded for the business data; during the data synchronization process, the auto-increment id is used to first find the largest id in the synchronized data, and then the data with id greater than the synchronized largest id is filtered out from the data source for synchronization; When synchronizing data with updated data in the data source, the data synchronization module is used to filter data with both the update date and creation date on the same day and synchronize them to the data warehouse; the data is pre-processed through the DW layer, and the latest data is obtained in chronological order to ensure that the data is consistent with the data source, and the data change information is recorded at the same time; For business data in the data source that is business data stored in files, the data synchronization module first compresses the business data and then transmits the business data to the data warehouse through the network; For business data with deletion behavior in the data source, the data synchronization module is used to add logical deletion fields to the business data at the business level; when the business data cannot have logical deletion fields added, the data synchronization module reads the database binlog log, synchronizes the primary key ID field, and then the data warehouse processes the business data and restores it to the dwd layer.

Images