Data processing system, method, computer device and readable storage medium

Abstract

The present application relates to the technical field of data query and processing, and specifically discloses a data processing system, method, computer device and readable storage medium. The system includes a data storage module, a Kylin multidimensional query component, a Druid storage connection pool and a data processing platform. The Kylin multidimensional query component is connected to the data storage module, the Druid storage connection pool and the data processing platform respectively; the data storage module is used to store data; the Kylin multidimensional query component is used to send the data query request to the Druid storage connection pool when receiving the data query request issued by the data processing platform, so as to instruct the Druid storage connection pool to retrieve the data corresponding to the data query request from the data storage module; the data processing platform is used to read the query results and process the query results. The Kylin on Druid architecture significantly improves the query and analysis speed of data.

Description

Technical Field

[0001] The present invention relates to the technical field of data query and processing, and in particular to a data processing system, method, computer equipment and readable storage medium. Background Art

[0002] With the development of the digital age, data has penetrated into all walks of life, and the demand for data query and processing is also expanding rapidly. However, due to the huge scale and multi-dimensional characteristics of data, the current data processing platform is often unable to quickly obtain the required data during the query and processing of data due to the performance limitations of various components, resulting in data delays, data asynchrony, and freezes, which is not conducive to data query and processing. Summary of the invention

[0003] Based on this, it is necessary to provide a data processing system, a data processing method, a computer device, a computer-readable storage medium and a computer program product to address the above problems.

[0004] In a first aspect, the present application provides a data processing system, the data processing system comprising a data storage module, a Kylin multi-dimensional query component, a Druid storage connection pool and a data processing platform, the Kylin multi-dimensional query component being connected to the data storage module, the Druid storage connection pool and the data processing platform respectively;

[0005] The data storage module is used to store data; the Kylin multi-dimensional query component is used to send the data query request to the Druid storage connection pool when receiving the data query request issued by the data processing platform, so as to instruct the Druid storage connection pool to retrieve the data corresponding to the data query request from the data storage module; the data processing platform is used to read the query results and process the query results.

[0006] In one of the embodiments, the data storage module includes a data collection component and a data source generation component, the data collection component is used to collect data, and the data source generation component is used to process the collected data to form a plurality of data sources.

[0007] In one of the embodiments, the data collection component is used to divide the collected data into multiple shared units, and the data in the same shared unit are distinguished by different identifiers.

[0008] In one of the embodiments, the data source generation component is used to filter the data in each of the shared units according to the identifier to form a plurality of data sources.

[0009] In one embodiment, the Kylin multidimensional query component includes a query server and a preprocessing server, the query server is respectively connected to the data processing platform and the Druid storage connection pool, and the preprocessing server is respectively connected to the data storage module and the Druid storage connection pool;

[0010] The query server is used to receive data query requests issued by the data processing platform, and forward the data query requests to the Druid storage connection pool, so that the Druid storage connection pool sends data retrieval requests to the data storage module. The preprocessing server is used to preprocess the data retrieved from the data storage module and send it to the Druid storage connection pool. The data processing platform reads the query results from the Druid storage connection pool via the query server.

[0011] In one of the embodiments, the preprocessing server is used to process the retrieved data in multiple dimensions, generate a Cuboid data file, and convert the Cuboid data file into a column storage format and send it to the Druid storage connection pool.

[0012] In one embodiment, the Druid storage connection pool includes a query node and a history node, both of which are connected to the Kylin multi-dimensional query component; the query node is used to receive the data query request and forward the data query request to the history node, and the history node retrieves the corresponding data from the data storage module according to the data query request.

[0013] In one of the embodiments, the data stored in the data storage module includes at least one of user behavior data, equity data, registration data, and achievement tables generated by marketing activities.

[0014] In a second aspect, the present application further provides a data processing method, which is applied to a data processing system, wherein the data processing system includes a data storage module, a Kylin multi-dimensional query component, a Druid storage connection pool, and a data processing platform; the data processing method includes:

[0015] When the Kylin multi-dimensional query component receives a data query request from the data processing platform, it sends the data query request to the Druid storage connection pool;

[0016] The Druid storage connection pool retrieves data corresponding to the data query request from the data storage module;

[0017] The data processing platform reads query results from the Druid storage connection pool.

[0018] In one embodiment, the data processing method further comprises the step of storing data in the data storage module:

[0019] Collecting data, and dividing the collected data into a plurality of shared units, wherein the data in the same shared unit are distinguished by different identifiers;

[0020] The data in each of the shared units is filtered according to the identifier to form and store a plurality of data sources.

[0021] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the above method when executed by a processor.

[0023] In a fifth aspect, the present application also provides a computer program product, which includes a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0024] The above-mentioned data processing system, data processing method, computer device, computer-readable storage medium and computer program product, when using the Kylin multidimensional query component to query data, cancels the use of the Kylin multidimensional query component's native HBase storage pool for data storage, and instead uses the Druid storage connection pool as the storage pool of the Kylin multidimensional query component. When the Kylin multidimensional query component receives a data query request, it sends the data query request to the Druid storage connection pool to instruct the Druid storage connection pool to retrieve the data corresponding to the data query request from the data storage module, and then the data processing platform reads the query results from the Druid storage connection pool via the Kylin multidimensional query component. Since the Druid storage connection pool uses columnar storage and inverted index, the filtering performance is better than HBase, and Druid has the characteristics of OLAP and good secondary aggregation capabilities, it significantly improves the data query capability, effectively reduces data delays, data asynchrony, and jamming, and other adverse phenomena, thereby helping to improve the data processing platform's analysis and processing speed of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the structure of a data processing system provided in one embodiment of the present application;

[0026] Figure 2A schematic diagram of the structure of a data processing system provided in one embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of a data storage module in a data processing system provided in one embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of a data processing system provided in one embodiment of the present application;

[0029] Figure 5 A flowchart of a data processing method provided in one embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0031] Description of reference numerals:

[0032] 100. Data storage module; 110. Data collection component; 120. Data source generation component; 200. Kylin multi-dimensional query component; 210. Query server; 220. Preprocessing server; 300. Druid storage connection pool; 310. Query node; 320. History node; 400. Data processing platform. DETAILED DESCRIPTION

[0033] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to understand the disclosure of the present invention more thoroughly and comprehensively.

[0034] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0035] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0037] As described in the background technology, the demand for data query and processing in various industries is increasing. For example, in the field of banking and finance, in order to maintain market share and expand customer areas, the banking industry is actively seeking new marketing methods to innovate marketing business using data. The demand is growing. The scale of marketing activity data is large, the structure is diverse and it has real-time characteristics. However, the existing data analysis platforms are only multifunctional tools for creating detailed graphics and displaying intuitive data. They have a single structure and weak analysis capabilities. For large data scales, they can only rely on the performance of local data warehouses. However, the performance of local data warehouses has limitations. It cannot pre-process data. In slightly complex query scenarios such as multi-table queries, the data analysis platform often freezes, and offline data analysis is relatively slow. In addition, due to the performance limitations of various components, the current data query and processing system cannot be updated in real time during actual data update and transmission, which further leads to data delays and data asynchrony, which is not conducive to the query of marketing data in practical applications.

[0038] In response to the above problems, the present application provides a data processing system, a data processing method, a computer device, a computer-readable storage medium, and a computer program product for improving the query and processing capabilities of data.

[0039] The data processing system, data processing method, computer device, computer-readable storage medium and computer program product provided in this application can be applied to fields that require data query and processing. This article only uses data query processing in the banking and financial field as an example to illustrate.

[0040] The following are explanations of the English terms that appear in this article:

[0041] Druid: is a data storage connection pool designed for high-performance slicing and user behavior data analysis on large data sets.

[0042] Apache Kylin: is a multidimensional query component that can perform deep and multidimensional processing on relevant database results.

[0043] Hive: It is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop.

[0044] Superset: is a "modern enterprise-level BI (business intelligence) web application" open sourced by Airbnb, which provides lightweight data query and visualization solutions for data analysis by creating and sharing dashboards.

[0045] Kafka: It was originally developed by Linkedin. It is a distributed, partitioned, multi-replica, multi-subscriber distributed logging system based on zookeeper coordination. It can be commonly used for web / nginx logs, access logs, message services, etc.

[0046] Cuboid: refers to all calculated data under a certain dimension combination in Kylin.

[0047] OLAP: On-Line Analytical Processing, provides users with simple and direct query results and supports complex analysis functions.

[0048] In one embodiment, a data processing system is provided, referring to Figure 1 The data processing system provided in this embodiment includes a data storage module 100, a Kylin multi-dimensional query component 200, a Druid storage connection pool 300 and a data processing platform 400. The Kylin multi-dimensional query component 200 is connected to the data storage module 100, the Druid storage connection pool 300 and the data processing platform 400 respectively. Among them:

[0049] The data storage module 100 is used to store data. The Kylin multidimensional query component 200 is used to send the data query request to the Druid storage connection pool 300 when receiving the data query request from the data processing platform 400, so as to instruct the Druid storage connection pool 300 to retrieve the data corresponding to the data query request from the data storage module 100. The data processing platform 400 is used to read the query results and process the query results.

[0050] Specifically, the data storage module 100 is mainly used to collect and store data generated by in-line marketing activities. During the storage process, the original data can be directly stored, or the original data can be integrated and processed before storage. The data stored in the data storage module 100 can be used as the data source of the entire data processing system, and can be retrieved from the data storage module 100 when inquiring.

[0051] Kylin multidimensional query component 200 is an open source distributed analysis engine that provides SQL query interfaces and multidimensional (OLAP) capabilities on Hadoop and Spark, and can query huge database tables in sub-seconds. Kylin multidimensional query component 200 can quickly respond to the set multidimensional data allocation method through pre-calculation, and can handle ultra-large-scale data sets and establish responses. Druid storage connection pool 300 tends to be real-time data analysis. It is an open source distributed data storage that combines the characteristics of analytical databases, time series databases, and search systems. It can handle data collection and analysis tasks on larger data sets, provide always-on real-time data ingestion and arbitrary data processing functions, and can effectively handle data queries with huge data volumes.

[0052] The Kylin on Druid architecture is adopted in this application, that is, the Druid storage connection pool 300 is used as the storage pool of the Kylin multi-dimensional query component 200, and the native storage engine HBase of the Kylin multi-dimensional query component 200 is cancelled. The data processing platform 400 provides a data query port. In actual applications, users can query data through the data processing platform 400. The data processing platform 400 sends a data query request to the Kylin multi-dimensional query component 200, and the Kylin multi-dimensional query component 200 sends the data query request to the Druid storage connection pool 300, so that the Druid storage connection pool 300 retrieves the data corresponding to the data query request from the data storage module 100. The data processing platform 400 then reads the retrieved data from the Druid storage connection pool 300 through the Kylin multi-dimensional query component 200 to complete the query, and can perform subsequent analysis and processing on the queried data.

[0053] Since the Druid storage connection pool 300 uses column storage and inverted index, the filtering performance is better than HBase, and Druid has the characteristics of OLAP and good secondary aggregation capabilities. Therefore, the Kylin on Druid architecture is adopted to significantly improve the query and analysis speed of data, effectively reduce data delays, data asynchrony, and jamming, which helps to optimize the data processing performance of the data processing platform 400.

[0054] In one embodiment, reference Figure 2 The data storage module 100 includes a data collection component 110 and a data source generation component 120. The data collection component 110 is used to collect data, and the data source generation component 120 is used to process the collected data to form a plurality of data sources.

[0055] Specifically, the data collection component 110 is responsible for collecting data generated by marketing activities, which may include at least one of user behavior data, equity data, registration data, and compliance tables generated by marketing activities. Among them, user behavior data refers to the PV, UV, button clicks, channel access data, etc. of the activities generated in the marketing activities. This part of the data belongs to the dimension after the user enters the activity page and does not participate in the activity task in depth; the equity data, registration data, and compliance tables generated by marketing activities refer to the activity registration data generated by the user entering the activity to participate in the task, the equity issued by the activity after completing the task, and the data for determining whether the user status meets the standard after completing the specified task steps. This part of the data belongs to the dimension after the user enters the activity to participate in the task.

[0056] The data collection component 110 may include a Kafka log system. In this embodiment, data logs may be formed by front-end embedding or interface parameter transmission and transmitted to the Kafka log system.

[0057] refer to Figure 3 In this embodiment, during the data collection process, in order to avoid creating a large number of topics in the Kafka log system, the collected data can be divided into several shared topics (i.e., shared units), and the data in the same shared unit are distinguished by different identifiers. For example, N topics are created in the Kafka log system, each topic contains several data, and each data is identified by a certain field for easy distinction. For example, topic 1 contains data a, data b, etc., data a is identified by identifier 1, and data b is identified by identifier 2. By using data identification, multiple data are written into one topic, which reduces the number of topics in the Kafka log system, improves the utilization rate of topics in the Kafka log system, and thus improves query efficiency.

[0058] The data source generation component 120 is used to receive the collected data and process it to obtain a plurality of data sources. Specifically, the data source generation component 120 can filter the data in each shared unit according to the identifier to form a plurality of data sources. Figure 3 The data source generation component 120 includes a filter and Hive. First, the data in each shared unit in the Kafka log system can be filtered according to the identifier through the filter and then enter Hive, thereby obtaining multiple different data sources. In order to reduce the complexity of manipulating the Kafka process, after entering Hive, the Kafka supervision management module, namely Kafka supervision, can be used to manage the data source formation process.

[0059] Among them, the Hive structure is a data warehouse structure, which is used to store all user behavior data. The metadata in Hive includes the name of the table, the columns and partitions of the table and their attributes, the attributes of the table (whether it is an external table, etc.), the directory where the table data is located, etc. Its advantage is that the scale of the cluster can be expanded freely, and generally there is no need to restart the service. This feature facilitates the subsequent expansion of marketing data without affecting the entire data transmission and storage process, and avoids the risk of service suspension and restart when data is regularized.

[0060] In one embodiment, reference Figure 4 The Kylin multi-dimensional query component 200 includes a query server 210 and a pre-processing server 220. The query server 210 is respectively connected to the data processing platform 400 and the Druid storage connection pool 300, and the pre-processing server 220 is respectively connected to the data storage module 100 and the Druid storage connection pool 300.

[0061] Among them, the query server 210 is used to receive data query requests issued by the data processing platform 400, and forward the data query requests to the Druid storage connection pool 300, so that the Druid storage connection pool 300 sends a data retrieval request to the data storage module 100. The preprocessing server 220 is used to preprocess the data retrieved from the data storage module 100 and send it to the Druid storage connection pool 300. The data processing platform 400 reads the query results from the Druid storage connection pool 300 via the query server 210.

[0062] The Kylin multidimensional query component 200 improves the query efficiency of the Kylin multidimensional query component 200 by setting different servers to perform different operations. Specifically, the preprocessing server 220 in the Kylin multidimensional query component 200 can be connected to Hive. After receiving the data retrieved from Hive, it can pre-calculate the data, use space to exchange time, and thus improve the utilization of space, and can quickly respond to the multidimensional data allocation method to improve the overall query speed.

[0063] In one embodiment, the preprocessing server 220 is used to process the retrieved data in multiple dimensions, generate Cuboid data files, and convert the Cuboid data files into column storage format and send them to the Druid storage connection pool 300. That is, the preprocessing server 220 first performs multi-dimensional processing on the retrieved data to obtain the Cuboid data file, and then converts it into a segment file corresponding to the Druid storage connection pool 300 and sends it to the Druid storage connection pool 300 for storage, thereby coordinating the Kylin multi-dimensional query component 200 with the Druid storage connection pool 300 to achieve efficient data query.

[0064] In one embodiment, reference Figure 4 The Druid storage connection pool 300 includes a query node 310 and a historical node 320, both of which are connected to the Kylin multi-dimensional query component 200; the query node 310 is used to receive data query requests and forward the data query requests to the historical node 320, and the historical node 320 retrieves the corresponding data from the data storage module 100 according to the data query request.

[0065] The Druid storage connection pool 300 is configured with multiple nodes, each of which is used to implement different operations, thereby improving internal operating efficiency. When the query server 210 in the Kylin multidimensional query component 200 receives a data query request sent by the data processing platform 400, it is forwarded to the query node 310 in the Druid storage connection pool 300, that is, the broker node. After parsing the request, the broker node finds the corresponding historical node 320, that is, the historical node, and distributes the request through the historical node. Finally, the data obtained from Hive is pre-calculated and processed by the pre-processing server 220 in the Kylin multidimensional query component 200 and then fed back to the historical node.

[0066] In one of the embodiments, the data processing platform 400 in this embodiment may select the superset data platform, which is a multifunctional tool for creating detailed data trend graphs and displaying intuitive data, supporting visualization of data by activity and subsequent data analysis functions. Selecting the superset data platform facilitates displaying data query results in the form of a visualization dashboard for marketing data, and is also conducive to data analysis tasks on the queried marketing data.

[0067] In one embodiment, the present application further provides a data processing method, which is applied to the aforementioned data processing system, wherein the data processing system includes a data storage module 100, a Kylin multi-dimensional query component 200, a Druid storage connection pool 300 and a data processing platform 400. Figure 5 The data processing method provided in this embodiment includes the following steps:

[0068] Step S200 , when the Kylin multi-dimensional query component 200 receives a data query request from the data processing platform 400 , it sends the data query request to the Druid storage connection pool 300 .

[0069] Step S400 : The Druid storage connection pool 300 retrieves data corresponding to the data query request from the data storage module 100 .

[0070] Step S600: The data processing platform 400 reads the query results from the Druid storage connection pool 300.

[0071] The Kylin on Druid architecture is adopted in this application, that is, the Druid storage connection pool 300 is used as the storage pool of the Kylin multi-dimensional query component 200, and the native storage engine HBase of the Kylin multi-dimensional query component 200 is cancelled. The data processing platform 400 provides a data query port. In actual applications, users can query data through the data processing platform 400. The data processing platform 400 sends a data query request to the Kylin multi-dimensional query component 200, and the Kylin multi-dimensional query component 200 sends the data query request to the Druid storage connection pool 300, so that the Druid storage connection pool 300 retrieves the data corresponding to the data query request from the data storage module 100. The data processing platform 400 then reads the retrieved data from the Druid storage connection pool 300 through the Kylin multi-dimensional query component 200 to complete the query, and can perform subsequent analysis and processing on the queried data.

[0072] Since the Druid storage connection pool 300 uses column storage and inverted index, the filtering performance is better than HBase, and Druid has the characteristics of OLAP and good secondary aggregation capabilities. Therefore, the Kylin on Druid architecture is adopted to significantly improve the query and analysis speed of data, effectively reduce data delays, data asynchrony, and jamming, which helps to optimize the data processing performance of the data processing platform 400.

[0073] In one of the embodiments, the data processing method provided in the embodiment further includes the step of storing data in the data storage module 100: collecting data, and dividing the collected data into a plurality of shared units, and distinguishing the data in the same shared unit by different identifiers;

[0074] The data in each shared unit is filtered according to the identifier to form and store several data sources.

[0075] The data processing method provided in this embodiment and the data processing system provided in the aforementioned embodiment belong to the same inventive concept. For the specific content of the data processing method, please refer to the corresponding description of the aforementioned data processing system, which will not be repeated here.

[0076] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0077] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0078] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0079] Step S200 , when the Kylin multi-dimensional query component 200 receives a data query request from the data processing platform 400 , it sends the data query request to the Druid storage connection pool 300 .

[0080] Step S400 : The Druid storage connection pool 300 retrieves data corresponding to the data query request from the data storage module 100 .

[0081] Step S600: The data processing platform 400 reads the query results from the Druid storage connection pool 300.

[0082] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0083] Collect data and divide the collected data into multiple shared units. The data in the same shared unit are distinguished by different identifiers.

[0084] The data in each shared unit is filtered according to the identifier to form and store several data sources.

[0085] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0086] Step S200 , when the Kylin multi-dimensional query component 200 receives a data query request from the data processing platform 400 , it sends the data query request to the Druid storage connection pool 300 .

[0087] Step S400 : The Druid storage connection pool 300 retrieves data corresponding to the data query request from the data storage module 100 .

[0088] Step S600: The data processing platform 400 reads the query results from the Druid storage connection pool 300.

[0089] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0090] Collect data and divide the collected data into multiple shared units. The data in the same shared unit are distinguished by different identifiers.

[0091] The data in each shared unit is filtered according to the identifier to form and store several data sources.

[0092] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0093] Step S200 , when the Kylin multi-dimensional query component 200 receives a data query request from the data processing platform 400 , it sends the data query request to the Druid storage connection pool 300 .

[0094] Step S400 : The Druid storage connection pool 300 retrieves data corresponding to the data query request from the data storage module 100 .

[0095] Step S600: The data processing platform 400 reads the query results from the Druid storage connection pool 300.

[0096] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0097] Collect data and divide the collected data into multiple shared units. The data in the same shared unit are distinguished by different identifiers.

[0098] The data in each shared unit is filtered according to the identifier to form and store several data sources.

[0099] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0100] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing system, characterized in that: The data processing system includes a data storage module, a Kylin multi-dimensional query component, a Druid storage connection pool and a data processing platform, and the Kylin multi-dimensional query component is connected to the data storage module, the Druid storage connection pool and the data processing platform respectively; The data storage module is used to store data; the Kylin multi-dimensional query component is used to send the data query request to the Druid storage connection pool when receiving the data query request issued by the data processing platform, so as to instruct the Druid storage connection pool to retrieve the data corresponding to the data query request from the data storage module; The data processing platform is used to read the query results and process the query results; Among them, the Kylin multi-dimensional query component includes a query server and a preprocessing server, the query server is respectively connected to the data processing platform and the Druid storage connection pool, and the preprocessing server is respectively connected to the data storage module and the Druid storage connection pool; The query server is used to receive data query requests issued by the data processing platform, and forward the data query requests to the Druid storage connection pool, so that the Druid storage connection pool sends a data retrieval request to the data storage module. The preprocessing server is used to process the data retrieved from the data storage module in multiple dimensions, generate a Cuboid data file, and convert the Cuboid data file into a column storage format and send it to the Druid storage connection pool. The data processing platform reads the query results from the Druid storage connection pool via the query server.

2. The data processing system according to claim 1, characterized in that The data storage module includes a data collection component and a data source generation component. The data collection component is used to collect data, and the data source generation component is used to process the collected data to form a plurality of data sources.

3. The data processing system according to claim 2, characterized in that: The data collection component is used to divide the collected data into multiple shared units, and the data in the same shared unit are distinguished by different identifiers.

4. The data processing system according to claim 3, characterized in that: The data source generation component is used to filter the data in each of the shared units according to the identifier to form a plurality of data sources.

5. The data processing system according to claim 1, characterized in that: The Druid storage connection pool includes query nodes and history nodes, and both the query nodes and the history nodes are connected to the Kylin multi-dimensional query component; The query node is used to receive the data query request and forward the data query request to the historical node. The historical node retrieves corresponding data from the data storage module according to the data query request.

6. The data processing system according to claim 1, characterized in that: The data stored in the data storage module includes at least one of user behavior data, equity data, registration data and achievement table generated by marketing activities.

7. A data processing method, characterized in that: Applied to a data processing system, the data processing system includes a data storage module, a Kylin multi-dimensional query component, a Druid storage connection pool and a data processing platform; The data processing method comprises: When the Kylin multi-dimensional query component receives a data query request from the data processing platform, it sends the data query request to the Druid storage connection pool; The Druid storage connection pool retrieves data corresponding to the data query request from the data storage module; The data processing platform reads the query results from the Druid storage connection pool; The Kylin multi-dimensional query component includes a query server and a pre-processing server; When the Kylin multi-dimensional query component receives a data query request from the data processing platform, it sends the data query request to the Druid storage connection pool, including: The query server receives a data query request sent by the data processing platform, and forwards the data query request to the Druid storage connection pool, so that the Druid storage connection pool sends a data retrieval request to the data storage module; After the Druid storage connection pool retrieves data corresponding to the data query request from the data storage module, the method includes: The preprocessing server processes the data retrieved from the data storage module in multiple dimensions to generate a Cuboid data file, and converts the Cuboid data file into a column storage format and sends it to the Druid storage connection pool; The data processing platform reads the query result from the Druid storage connection pool, including: The data processing platform reads the query results from the Druid storage connection pool via the query server.

8. The data processing method according to claim 7, characterized in that: The data processing method further comprises the step of storing data in the data storage module: Collecting data, and dividing the collected data into a plurality of shared units, wherein the data in the same shared unit are distinguished by different identifiers; The data in each of the shared units is filtered according to the identifier to form and store a plurality of data sources.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 7 or 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 or 8 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 7 or 8 are implemented.

Images