Data analysis processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data analysis processing method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of data processing. The analysis processing method comprises the steps that original data are accessed from a data source according to a message queue, and the original data comprise real-time data and offline data; performing real-time analysis processing on the real-time data according to a stream processing technology; and performing off-line analysis processing on the off-line data according to an on-line analytical processing (OLAP) technology and the materialized view. The problems of low resource utilization rate, high operation and maintenance complexity, high cost, high data delay and data storage limitation in related technologies are at least solved. The method is suitable for data analysis processing scenes.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data analysis and processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of big data technology, enterprises and organizations face the challenge of processing and analyzing massive amounts of data. Traditional data warehouses (DW) and online analytical processing (OLAP) systems primarily process static data, while modern enterprises require both real-time and offline analytics capabilities to meet diverse business needs. Real-time data analytics allows enterprises to respond quickly to market changes and user behavior, while offline analytics provides in-depth insights and decision support. However, the integration of real-time and offline analytics within a data warehouse often...

[0003] The following problems exist:

[0004] 1) Implementing and maintaining a real-time or offline data warehouse is very complex and costly, requiring significant upfront investment in infrastructure, software, and technical personnel. Continuous maintenance of data management performance increases the overall operating cost of the data warehouse.

[0005] 2) Traditional offline analysis generally uses the T+1 method to calculate the data of the previous day, which has a high data analysis latency. Even if real-time analysis is used, there is still a problem that the historical baseline cannot be calculated.

[0006] 3) Massive amounts of real-time data cannot be directly stored in a data warehouse. They are generally stored in Kafka, while offline data is mostly stored on HDFS (Hadoop Distributed File System).

[0007] 4) Offline analysis and processing primarily provides basic data collection and query capabilities, performing analysis and processing only on offline data. Therefore, existing technologies suffer from low resource utilization, high operational complexity, high cost, high data latency, and data storage limitations. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a data analysis and processing method, apparatus, electronic device, and computer-readable storage medium. This method can realize rapid and effective analysis and processing of real-time and offline data, ensuring the timeliness of data analysis and processing while improving the performance of massive data storage, analysis, query, and deep mining, saving resources and maintenance costs, and improving data utilization.

[0009] In a first aspect, the present invention provides a data analysis and processing method, comprising: accessing raw data from a data source according to a message queue, wherein the raw data includes real-time data and offline data; performing real-time analysis and processing on the real-time data according to stream processing technology; and performing offline analysis and processing on the offline data according to Online Analytical Processing (OLAP) technology and materialized views.

[0010] Preferably, the raw data further includes business information data, and the step of accessing the raw data from the data source according to the message queue specifically includes: using Kafka as the message queue to access real-time data, offline data and business information data from the data source; and storing the business information data in a data warehouse.

[0011] Preferably, the step of performing real-time analysis and processing on real-time data according to stream processing technology specifically includes: performing real-time analysis and processing on real-time data according to predefined rules of Apache Flink technology and the Complex Event Processing Library (CEP) to obtain real-time data after real-time analysis and processing; extracting business information data from the data warehouse; and associating and identifying the real-time data after real-time analysis and processing with the business information data to obtain associated and identified real-time data.

[0012] Preferably, after performing real-time analysis and processing on the real-time data according to the stream processing technology, and before performing offline analysis and processing on the offline data according to the Online Analytical Processing (OLAP) technology and materialized views, the data analysis and processing method further includes: storing the associated and identified real-time data into a data warehouse.

[0013] Preferably, the offline analysis and processing of offline data based on Online Analytical Processing (OLAP) technology and materialized views specifically includes: extracting real-time data after association and identification from the data warehouse; fusing offline data and real-time data after association and identification based on OLAP technology to obtain topic wide tables for each business; and aggregating and statistically analyzing the topic wide tables for each business based on materialized views to obtain offline data after offline analysis and processing.

[0014] Preferably, after extracting the associated and identified real-time data from the data warehouse, and before fusing the offline data and the associated and identified real-time data according to OLAP technology to obtain the topic wide tables for each business, the data analysis and processing method further includes: performing data preprocessing on the offline data and the associated and identified real-time data, wherein the data preprocessing includes, but is not limited to: normalization processing.

[0015] Preferably, after performing offline analysis processing on the offline data using Online Analytical Processing (OLAP) technology and materialized views, the data analysis processing method further includes: scheduling preset computing resources using Remote Procedure Call (RPC) technology to perform advanced data analysis on the offline data after offline analysis processing, wherein the preset computing resources include, but are not limited to, the calculation scripts of baseline models, and the advanced data analysis includes, but is not limited to, model analysis.

[0016] Secondly, the present invention also provides a data analysis and processing apparatus, comprising: an access module, a real-time processing module, and an offline processing module. The access module is used to access raw data from a data source according to a message queue, wherein the raw data includes real-time data and offline data. The real-time processing module is connected to the access module and is used to perform real-time analysis and processing on the real-time data according to stream processing technology. The offline processing module is connected to the access module and is used to perform offline analysis and processing on the offline data according to Online Analytical Processing (OLAP) technology and materialized views.

[0017] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the data analysis and processing method provided in the first aspect above.

[0018] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the data analysis and processing method provided in the first aspect above.

[0019] This invention provides a data analysis and processing method, apparatus, electronic device, and computer-readable storage medium. It receives real-time and offline data from a data source via a message queue, performs real-time analysis and processing on the real-time data using stream processing technology, and performs offline analysis and processing on the offline data using OLAP technology and materialized views. Therefore, this invention enables rapid and effective analysis and processing of both real-time and offline data, ensuring the timeliness of data analysis and processing while improving the performance of massive data storage, analysis, querying, and deep mining, saving resources and maintenance costs, and increasing data utilization. Attached Figure Description

[0020] Figure 1 This is a flowchart of a data analysis and processing method according to Embodiment 1 of the present invention;

[0021] Figure 2 This is a schematic diagram of the structure of a data analysis and processing system according to Embodiment 2 of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of a data analysis and processing device according to Embodiment 3 of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0024] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0025] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0027] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0028] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0029] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0030] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0031] Example 1:

[0032] like Figure 1 As shown, this embodiment provides a data analysis and processing method. The data analysis and processing method includes:

[0033] S101: Access raw data from the data source according to the message queue. The raw data includes real-time data and offline data.

[0034] Optionally, the raw data may also include business information data.

[0035] In this embodiment, the data in the data source includes, but is not limited to, log data, traffic data, and business information data. Log data includes Syslog, server logs, and operating system logs; traffic data includes DB (Database) auditing and API (Application Programming Interface) auditing. Real-time data generated in log data and traffic data is considered real-time data; non-real-time data generated in log data and traffic data is considered offline data. Business information data is mainly used to supplement real-time and offline data, primarily consisting of dimension table data, such as user account information tables and email account information tables. Therefore, the data in the data source can be classified into real-time data, offline data, and business information data to be accessed and analyzed as raw data. The data source can be considered as various real-time data streams or various business systems. Real-time data streams include, but are not limited to, message queues, log data streams, and sensor data streams; business systems include, but are not limited to, ERP (Enterprise Resource Planning) systems and CRM (Customer Relationship Management) systems.

[0036] Specifically, S101: Access raw data from the data source according to the message queue, where the raw data includes real-time data and offline data, including steps S1011-S1012:

[0037] The S1011 uses Kafka as a message queue to access real-time data, offline data, and business information data from data sources.

[0038] In this embodiment, Kafka is used as the message queue middleware to enable the access of real-time data, offline data, and business information data from the data source. The message queue middleware can also be RabbitMQ or Apache ActiveMQ. This embodiment also supports the collection of business information data via API interfaces.

[0039] S1012 stores business information data in a data warehouse.

[0040] In this embodiment, a hybrid real-time / offline OLAP data warehouse is obtained by integrating real-time analytical processing (ARPA) technology, offline ARPA technology, and data warehousing. This embodiment uses stream processing as the real-time ARPA technology and OLAP technology and materialized views as the offline ARPA technology. Compared to existing real-time and offline data warehouses, the hybrid real-time / offline OLAP can process both real-time and offline data. The hybrid real-time / offline OLAP data warehouse includes a real-time processing layer, an offline processing layer, and a data warehouse. The data warehouse includes an ODS (Operational Data Store) raw data layer, a DWD (Data Warehouse Detail) data layer, a DWM (Data Warehouse Middle) data intermediate layer, a DWS (Data Warehouse Summary) summary data layer, and an ADS (Application Data Store) data application layer.

[0041] After receiving real-time data, offline data, and business information data from the data source, on the one hand, these data are fed into the real-time processing layer and the offline processing layer for real-time and offline analysis and processing, respectively. On the other hand, the real-time data and offline data are stored in the ODS (Original Data Storage) layer of the data warehouse, while the business information data is stored in the DWD (Detailed Data Storage) layer of the data warehouse in the form of DIM (Dimensional Data Storage) tables. This embodiment integrates real-time analysis and processing technology, offline analysis and processing technology, and data warehousing to obtain a hybrid real-time and offline OLAP data warehouse. This supports the integration of real-time and offline data, enabling enterprises to utilize their data assets more effectively without the need to deploy separate data warehouses for real-time and offline data, saving on data warehouse deployment and maintenance costs and resources. The data warehouse not only provides powerful storage capabilities but also offers efficient data security measures, helping to protect sensitive and confidential information and prevent unauthorized access and data leakage.

[0042] The ODS (Original Data System) raw layer primarily stores and manages raw data collected from various source systems. This raw data typically undergoes no cleaning or transformation, retaining only its original format and content. As the first step in the data cleaning, transformation, and loading (ETL) process—the process of extracting, transforming, and loading data from the source to the destination—the ODS raw layer ensures that all subsequent analyses and reports are based on the same, unified data definition. The DWD (Detailed Data Layer) primarily stores detailed business information data, including detailed business information data after initial cleaning and transformation. This business information data is stored and managed in the form of DIM (Dimension) tables, which are more structured and facilitate subsequent analysis and querying, supporting advanced analysis and complex queries. The DWM (Intermediate Data Layer) primarily stores deeply processed data. The DWM layer mainly performs aggregation and statistical indicator operations, suitable for complex analysis and report generation, reducing the data processing burden on upper layers (such as DWS and ADS) and improving analytical efficiency and performance. The DWS (Data Summary Layer) is primarily used to store summary data. The data in the DWS layer is a highly aggregated and summarized version of the underlying data, mainly used for fast querying and report generation. It provides quick access and simple query capabilities, suitable for decision support and management reporting. The ADS (Application Data Layer) is primarily used to support specific business applications and analytical needs. The data in the ADS layer is optimized and customized for specific applications or analytical scenarios, directly supporting business applications and analysis, including but not limited to data visualization and machine learning model training.

[0043] S102 performs real-time analysis and processing of real-time data based on stream processing technology.

[0044] Specifically, S102: Based on stream processing technology, real-time analysis and processing are performed on the real-time data, including steps S1021-S1023:

[0045] S1021, based on the predefined rules of Apache Flink technology and the Complex Event Processing Library (CEP), performs real-time analysis and processing on real-time data to obtain the real-time data after real-time analysis and processing.

[0046] In this embodiment, real-time data analysis primarily relies on stream processing technologies, such as Apache Kafka and Apache Flink, which can process and analyze large volumes of real-time data streams. Offline data analysis, on the other hand, typically relies on batch processing technologies, such as Apache Hadoop and Apache Spark, which can handle large amounts of offline data. For real-time data analysis, Apache Flink is an open-source stream processing framework capable of processing real-time data streams and providing rich APIs and algorithms for data processing and analysis. Real-time data is analyzed using Flink on YARN (i.e., publishing Flink tasks on YARN, with YARN handling resource scheduling and management during task execution). This analysis includes, but is not limited to, predefined rule analysis and CEP (Complex Event Processing) analysis. This embodiment leverages Flink's real-time data processing capabilities to perform real-time analysis and processing of real-time data, utilizing Flink's high throughput and low latency to improve the efficiency of real-time data analysis and processing, and enabling rapid response to market changes and user behavior.

[0047] S1022, Extract business information data from the data warehouse.

[0048] S1023, associate and label the real-time data after real-time analysis and processing with business information data to obtain the associated and labeled real-time data.

[0049] In this embodiment, the DIM dimension tables of real-time data and business information data after real-time analysis and processing are correlated and labeled. The DIM dimension table of the business information data is used as an auxiliary table to supplement the field content of the real-time data. For example, for a user calling an API interface, the real-time data only contains the user ID, API address, and calling method information. The correlation and labeling process combines the user information table, API address table, and other external tables of the business information data to supplement the missing information. The real-time data is labeled with the username "Zhang San" and the IP address of the application system to which the API belongs. This embodiment, by correlating and labeling the real-time data with the DIM dimension tables of the business information data, can supplement and enrich the real-time data, improving its completeness and accuracy.

[0050] S103, performs offline analysis and processing on offline data based on Online Analytical Processing (OLAP) technology and materialized views.

[0051] Specifically, S103: Based on Online Analytical Processing (OLAP) technology and materialized views, perform offline analysis processing on the offline data, including steps S1031-S1033:

[0052] S1031, extract real-time data after association and identification from the data warehouse.

[0053] In this embodiment, after associating and identifying the real-time data after real-time analysis and processing with business information data, the associated and identified real-time data is stored in the DWM data middleware layer of the data warehouse for use by the offline processing layer. It is then merged with offline data to complete the field content of the offline data and generate thematic wide tables under each business.

[0054] S1032, based on OLAP technology, integrates offline data and real-time data after association and identification to obtain thematic wide tables for each business.

[0055] In this embodiment, taking the offline processing layer as an OLAP engine as an example, the process is similar to that of real-time data after association and identification analysis. Considering that building a topic wide table solely for offline data may not provide sufficient information, it is usually necessary to combine it with business information data for association and tagging to make the generated topic wide table more comprehensive and readable. To further facilitate comprehensive decision-making by enterprises using both real-time and offline data, this embodiment integrates offline data with the associated and tagged real-time data to generate a topic wide table that includes business information, real-time information, and offline information. For example, offline data may only contain user IDs, but by associating offline data, user names, phone numbers, email addresses, and VPN numbers can be added and stored together in the topic wide table for convenient subsequent data analysis and calculation queries. This embodiment uses the DIM dimension table in the business information data to associate and tag offline data, supplementing and enriching the offline data, improving its completeness and accuracy. Furthermore, by integrating offline and real-time data, it fully combines the advantages of high throughput of offline data and high timeliness of real-time data, facilitating comprehensive decision-making by enterprises using both real-time and offline data.

[0056] S1033: Based on the materialized view, aggregate and statistically analyze the wide tables of various business functions to obtain offline data after offline analysis and processing.

[0057] In this embodiment, materialized views can be used to perform statistical analysis based on long historical periods, such as the data statistics for the most recent six months. Materialized views also allow users to pre-calculate and store query results, thereby accelerating query speed. This embodiment uses materialized views to aggregate and statistically analyze thematic wide tables under various business functions, providing in-depth insights and decision support, facilitating subsequent queries and decisions on various business thematic wide tables, and improving the efficiency of querying and decision-making.

[0058] Optionally, after extracting the associated and identified real-time data from the data warehouse in S1031, and before fusing the offline data and the associated and identified real-time data according to OLAP technology to obtain the topic wide tables under each business in S1032, the data analysis and processing method further includes: performing data preprocessing on the offline data and the associated and identified real-time data, wherein the data preprocessing includes, but is not limited to: normalization processing.

[0059] In this embodiment, considering the large range and scale of both offline data and the associated and labeled real-time data, normalization is used to first convert the offline data and the associated and labeled real-time data into a unified and comparable range before merging them to obtain thematic wide tables for each business. This embodiment normalizes the offline data and the associated and labeled real-time data, unifying the data range and scale, making the data easier to process and analyze, and improving the efficiency of offline analysis and processing.

[0060] Optionally, after performing real-time analysis and processing on the real-time data according to stream processing technology in S102, and before performing offline analysis and processing on the offline data according to Online Analytical Processing (OLAP) technology and materialized views in S103, the data analysis and processing method further includes storing the associated and identified real-time data into a data warehouse.

[0061] In this embodiment, before fusing offline data and associated and labeled real-time data, the associated and labeled real-time data can also be aggregated and statistically analyzed through materialized views to facilitate subsequent querying and decision-making of the real-time data after real-time analysis and processing.

[0062] Optionally, in S103: After performing offline analysis on the offline data based on Online Analytical Processing (OLAP) technology and materialized views, the data analysis and processing methods further include:

[0063] S104. Based on Remote Procedure Call (RPC) technology, schedule preset computing resources to perform advanced data analysis on the offline data after offline analysis and processing. The preset computing resources include, but are not limited to, the calculation scripts for baseline models, and the advanced data analysis includes, but is not limited to, model analysis.

[0064] In this embodiment, RPC (Remote Procedure Call) technology is used to schedule preset computing resources to implement preset computing logic and perform advanced data analysis on the data in the data warehouse. For example, users can introduce Python machine learning libraries to perform in-depth analysis by loading offline data processed from the data warehouse. The data in the data warehouse includes offline data processed from offline analysis and real-time data after association and labeling. Preset computing resources include SQL (Structured Query Language), Python, or Java scripts. Preset computing logic includes, but is not limited to, baseline-type model computing logic. Baseline-type model computing logic is mainly based on the behavioral baselines of long-term computing entities. For example, by calculating the access frequency and access patterns of each user or the API call patterns, a baseline for each user is constructed, and then the baseline for each department is calculated to obtain baseline values ​​for various dimensions. This embodiment uses RPC technology to customize and introduce advanced data analysis computing resources to integrate and be compatible with different analysis scenarios, providing data mining and machine learning capabilities. This allows enterprises to flexibly conduct data analysis according to their own business needs, thereby improving decision-making efficiency and business value.

[0065] This embodiment provides a data analysis and processing method that accesses real-time and offline data from a data source via a message queue. Real-time data is analyzed and processed in real-time using stream processing technology, while offline data is analyzed and processed offline using OLAP technology and materialized views. This achieves rapid and effective analysis and processing of both real-time and offline data, ensuring timeliness while improving the performance of massive data storage, analysis, querying, and deep mining, saving resources and maintenance costs, and increasing data utilization. Furthermore, by integrating real-time analysis and processing technology, offline analysis and processing technology, and a data warehouse, a hybrid real-time / offline OLAP data warehouse is obtained. This supports the integration of real-time and offline data, enabling enterprises to utilize their data assets more effectively without the need for separate data warehouses for real-time and offline data, saving deployment and maintenance costs and resources. The data warehouse not only provides powerful storage capabilities but also offers efficient data security measures, helping to protect sensitive and confidential information and prevent unauthorized access and data leakage. Utilizing Flink's real-time data processing capabilities, real-time data is analyzed and processed in real-time, leveraging Flink's high throughput and low latency characteristics to improve the efficiency of real-time data analysis and processing, and enabling rapid response to market changes and user behavior. By associating and tagging real-time data with the DIM dimension tables of business information data, real-time data can be supplemented and enriched, improving its completeness and accuracy. Similarly, by associating and tagging offline data with the DIM dimension tables in business information data, and completing the fields in offline data, offline data can be supplemented and enriched, improving its completeness and accuracy. Furthermore, the fusion of offline and real-time data fully leverages the advantages of high throughput in offline data and high timeliness in real-time data, facilitating comprehensive decision-making for enterprises using both real-time and offline data. Materialized views aggregate and statistically analyze thematic wide tables under various business functions, providing in-depth insights and decision support, facilitating subsequent queries and decisions on various business thematic wide tables, and improving query and decision-making efficiency. Normalizing offline data and the associated and tagged real-time data, by unifying the data scope and scale, makes the data easier to process and analyze, improving the efficiency of offline analysis and processing. Through RPC technology, advanced data analysis computing resources can be customized and introduced to integrate and be compatible with different analysis scenarios, providing data mining and machine learning capabilities. This allows enterprises to flexibly conduct data analysis according to their own business needs, thereby improving decision-making efficiency and business value.

[0066] Example 2:

[0067] like Figure 2 As shown, this embodiment provides a data analysis and processing system, including an access layer, a real-time processing layer, an offline processing layer, a data warehouse, and an application layer.

[0068] The access layer is used to access raw data from the data source based on the message queue. The raw data includes real-time data, offline data, and business information data.

[0069] In this embodiment, the business information data is... Figure 2 The business data in the data source includes, but is not limited to, log data, traffic data, and business information data. Log data includes Syslog, server logs, and operating system logs; traffic data includes database auditing and API auditing. Real-time data refers to data generated in real-time from log data and traffic data; offline data refers to data generated in non-real-time from log data and traffic data. Business information data is mainly used to supplement real-time and offline data, primarily consisting of dimension tables such as user account information tables and email account information tables. Therefore, the data in the data source can be classified into real-time data, offline data, and business information data to be used as raw data for access and analysis. The data source can be considered as various real-time data streams or various business systems. Real-time data streams include, but are not limited to, message queues, log data streams, and sensor data streams; business systems include, but are not limited to, ERP systems and CRM systems.

[0070] Kafka is used as the message queue middleware to enable the ingestion of real-time data, offline data, and business information data from data sources. Other message queue middleware options include RabbitMQ and Apache ActiveMQ. This embodiment also supports the collection of business information data via API interfaces.

[0071] The real-time processing layer is used to process real-time data in real time using stream processing techniques.

[0072] In this embodiment, Apache Flink, an open-source stream processing framework, is used for real-time data analysis and processing. It can handle real-time data streams and provides rich APIs and algorithms for data processing and analysis. The Flink on YARN approach is used to analyze and process real-time data, including but not limited to predefined rule analysis and CEP analysis. This embodiment leverages Flink's real-time data processing capabilities to perform real-time analysis and processing on real-time data. By utilizing Flink's high throughput and low latency characteristics, it improves the efficiency of real-time data analysis and processing, and enables rapid response to market changes and user behavior.

[0073] The DIM dimension tables of real-time data and business information data, after real-time analysis and processing, are correlated and labeled. The DIM dimension tables of the business information data serve as auxiliary tables to supplement the field content of the real-time data. For example, for a user calling an API interface, the real-time data only contains the user ID, API address, and calling method information. The correlation and labeling process combines the user information table, API address table, and other external tables from the business information data to supplement the missing information. For instance, the real-time data is labeled with the username "Zhang San" and the IP address of the application system to which the API belongs. This embodiment, by correlating and labeling the real-time data with the DIM dimension tables of the business information data, can supplement and enrich the real-time data, improving its completeness and accuracy.

[0074] After associating and identifying the real-time data after real-time analysis and processing with business information data, the associated and identified real-time data is stored in the DWM data middleware layer of the data warehouse for use by the offline processing layer. It is then merged with offline data to complete the field content of the offline data and generate thematic wide tables for each business.

[0075] The offline processing layer is used to process offline data based on OLAP technology and materialized views.

[0076] In this embodiment, the offline processing layer is... Figure 2 Taking the offline processing layer as an example, the OLAP engine in this example is similar to the process of real-time data after correlation and identification analysis. Considering that building a topic wide table solely for offline data may not provide sufficient information, it is often necessary to combine it with business information data for correlation and tagging to make the generated topic wide table more comprehensive and readable. This embodiment further facilitates comprehensive decision-making by enterprises using both real-time and offline data. The OLAP engine integrates offline data with the correlated and identified real-time data to generate a topic wide table that includes business information, real-time information, and offline information. For example, offline data may only contain user IDs, but by correlating the offline data, user names, phone numbers, email addresses, and VPN numbers can be added and stored together in the topic wide table for convenient subsequent data analysis and calculation queries. This embodiment uses the DIM dimension table in the business information data to correlate and tag offline data, supplementing and enriching the offline data, improving its completeness and accuracy. Furthermore, by integrating offline and real-time data, it fully combines the advantages of high throughput of offline data and high timeliness of real-time data, facilitating comprehensive decision-making by enterprises using both real-time and offline data.

[0077] Materialized views can be used to perform statistical analysis based on long historical periods, such as the data from the most recent six months. Materialized views also allow users to pre-calculate and store query results, thereby accelerating query speed. This embodiment uses materialized views to aggregate and statistically analyze wide tables under various business functions, providing in-depth insights and decision support, facilitating subsequent queries and decisions on these wide tables, and improving query and decision-making efficiency.

[0078] Considering the large range and scale of both offline and real-time data after association and identification, normalization is used to first convert the offline and real-time data into a unified and comparable range before merging them to obtain thematic wide tables for each business. This embodiment normalizes both offline and real-time data after association and identification, unifying the data range and scale to make the data easier to process and analyze, thus improving the efficiency of offline analysis and processing.

[0079] Before integrating offline data and the associated and labeled real-time data, this embodiment can also aggregate and statistically analyze the associated and labeled real-time data through materialized views to facilitate subsequent querying and decision-making on the real-time data after real-time analysis and processing.

[0080] A data warehouse is used to store real-time data, offline data, business information data, real-time data after real-time processing, and offline data after offline processing. The data warehouse includes the ODS (Original Data Storage) layer, the DWD (Detailed Data Storage) layer, the DWM (Intermediate Data Storage) layer, the DWS (Summary Data Storage) layer, and the ADS (Application Data Storage) layer.

[0081] In this embodiment, after real-time data, offline data, and business information data are accessed from the data source, on the one hand, the real-time data, offline data, and business information data are accessed to the real-time processing layer and the offline processing layer for real-time analysis and processing and offline analysis and processing; on the other hand, the real-time data and offline data are stored in the ODS data raw layer of the data warehouse, and the business information data is stored in the DWD data detail layer of the data warehouse in the form of DIM dimension tables.

[0082] The ODS (Original Data System) raw data layer primarily stores and manages raw data collected from various source systems. This raw data typically undergoes no cleaning or transformation, retaining only its original format and content. As the first step in the ETL (Extract, Transform, Load) process, the ODS raw data layer ensures that all subsequent analyses and reports are based on the same, unified data definition. The DWD (Detailed Data Layer) primarily stores detailed business information data, including pre-cleaned and transformed data. This business information data is stored and managed in the form of DIM (Dimensional Data Objects) tables, providing a more structured structure that facilitates subsequent analysis and querying, supporting advanced analysis and complex queries. The DWM (Intermediate Data Layer) primarily stores deeply processed data. The DWM layer mainly performs aggregation and statistical indicator operations, suitable for complex analysis and report generation, reducing the data processing burden on upper layers (such as DWS and ADS) and improving analytical efficiency and performance. The DWS (Summary Data Layer) primarily stores summary data, which is the result of highly aggregated and summarized lower-level data. It is primarily used for fast querying and report generation, providing quick access and simple query capabilities, suitable for decision support and management reporting. The ADS data application layer is primarily used to support specific business applications and analytical needs. The data in the ADS layer is optimized and customized for specific applications or analytical scenarios, directly supporting business applications and analysis, including but not limited to data visualization and machine learning model training.

[0083] The application layer is used to schedule preset computing resources based on RPC technology to perform advanced data analysis on offline data after offline analysis and processing.

[0084] In this embodiment, RPC technology is used to schedule preset computing resources to implement preset computing logic for advanced data analysis of data in the data warehouse. For example, users can introduce Python machine learning libraries to perform in-depth analysis by loading offline data processed from the data warehouse. The data in the data warehouse includes offline data processed from offline analysis and real-time data after association and labeling. Preset computing resources include SQL, Python, or Java scripts, and preset computing logic includes, but is not limited to, baseline-type model computing logic. Baseline-type model computing logic is mainly based on the behavioral baselines of long-term computing entities. For example, by calculating the access frequency and access patterns of each user or the API call patterns, a baseline for each user is constructed, and then the baseline for each department is calculated to obtain baseline values ​​for various dimensions. This embodiment uses RPC technology to customize and introduce advanced data analysis computing resources to integrate and be compatible with different analysis scenarios, providing data mining and machine learning capabilities. This enables enterprises to flexibly conduct data analysis according to their own business needs, thereby improving decision-making efficiency and business value.

[0085] This embodiment provides a data analysis and processing system that integrates real-time analysis and processing technology, offline analysis and processing technology, and data warehousing to achieve hybrid analysis, processing, and storage of real-time and offline data. It supports the integration of real-time and offline data, enabling enterprises to utilize their data assets more effectively without the need for separate data warehouses for real-time and offline data, thus saving deployment and maintenance costs and resources. The data warehouse not only provides powerful storage capabilities but also offers efficient data security measures, helping to protect sensitive and confidential information and prevent unauthorized access and data leakage. Furthermore, leveraging Flink's real-time data processing capabilities, real-time analysis and processing of real-time data is performed, utilizing Flink's high throughput and low latency characteristics to improve the efficiency of real-time data analysis and processing, and enabling rapid response to market changes and user behavior. By associating and tagging real-time data with the DIM dimension tables of business information data, real-time data can be supplemented and enriched, improving its completeness and accuracy. By linking and tagging offline data through DIM dimension tables in business information data, and supplementing the fields of offline data, the completeness and accuracy of offline data can be improved. This also allows for the fusion of offline and real-time data, fully combining the advantages of high throughput of offline data and high timeliness of real-time data, facilitating comprehensive decision-making for enterprises using both real-time and offline data. Materialized views aggregate and statistically analyze thematic wide tables under various business functions, providing in-depth insights and decision support, facilitating subsequent queries and decisions on various business thematic wide tables, and improving query and decision-making efficiency. Normalization of offline data and linked and tagged real-time data, by unifying the scope and scale of data, makes the data easier to process and analyze, improving the efficiency of offline analysis and processing. Through RPC technology, customized advanced data analysis computing resources can be introduced to integrate and be compatible with different analysis scenarios, providing data mining and machine learning capabilities. This allows enterprises to flexibly conduct data analysis according to their own business needs, thereby improving decision-making efficiency and business value.

[0086] Example 3:

[0087] like Figure 3 As shown, this embodiment provides a data analysis and processing device, including: an access module 31, a real-time processing module 32, and an offline processing module 33. The access module 31 is used to access raw data from a data source according to a message queue, wherein the raw data includes real-time data and offline data. The real-time processing module 32 is connected to the access module 31 and is used to perform real-time analysis and processing on the real-time data according to stream processing technology. The offline processing module 33 is connected to the access module 31 and is used to perform offline analysis and processing on the offline data according to Online Analytical Processing (OLAP) technology and materialized views.

[0088] Specifically, the raw data also includes business information data. The access module 31 includes an access unit 311 and a first storage unit 312. The access unit 311 is used to use Kafka as a message queue to access real-time data, offline data and business information data from the data source. The first storage unit 312 is used to store the business information data in the data warehouse.

[0089] Specifically, the real-time processing module 32 includes: a real-time processing unit 321, a first extraction unit 322, and an association and identification unit 323. The real-time processing unit 321 is used to perform real-time analysis and processing on real-time data according to the predefined rules of Apache Flink technology and the Complex Event Processing Library (CEP) to obtain real-time data after real-time analysis and processing. The first extraction unit 322 is used to extract business information data from the data warehouse. The association and identification unit 323 is used to associate and identify the real-time data after real-time analysis and processing with the business information data to obtain the associated and identified real-time data.

[0090] Optionally, the data analysis and processing apparatus further includes a storage module 34 for storing the associated and identified real-time data into a data warehouse.

[0091] Specifically, the offline processing module 33 includes: a second extraction unit 331, a fusion unit 332, and an aggregation and statistics unit 333. The second extraction unit 331 is used to extract real-time data after association and identification from the data warehouse. The fusion unit 332 is used to fuse offline data and real-time data after association and identification according to OLAP technology to obtain thematic wide tables under each business. The aggregation and statistics unit 333 is used to aggregate and statistically analyze thematic wide tables under each business according to materialized views to obtain offline data after offline analysis and processing.

[0092] Optionally, the offline processing module 33 further includes a data preprocessing module 334, used to preprocess the offline data and the associated and identified real-time data, wherein the data preprocessing includes, but is not limited to, normalization processing.

[0093] Optionally, the data analysis and processing device further includes: a data analysis module 35, used to schedule preset computing resources according to remote procedure call (RPC) technology, and perform advanced data analysis on the offline data after offline analysis and processing, wherein the preset computing resources include, but are not limited to: the calculation script of the baseline model, and the advanced data analysis includes, but is not limited to: model analysis.

[0094] Understandably, the data analysis and processing device provided above executes the data analysis and processing method corresponding to Embodiment 1 provided above. Therefore, the beneficial effects it can achieve can be referred to the data analysis and processing method of Embodiment 1 above, and will not be repeated here.

[0095] Example 4:

[0096] This embodiment also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to implement the data analysis and processing method in Embodiment 1 above.

[0097] Example 5:

[0098] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the data analysis and processing method in Embodiment 1 above.

[0099] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method of analyzing data, characterized by, The method comprises the following steps: accessing raw data from a data source according to a message queue, wherein the raw data comprises real-time data and offline data; performing real-time analysis and processing on the real-time data according to a stream processing technology; performing offline analysis and processing on the offline data according to an online analytical processing (OLAP) technology and a materialized view.

2. The data analysis method according to claim 1, characterized by, The raw data further comprises business information data, The step of accessing the raw data from the data source according to the message queue specifically comprises the following steps: accessing real-time data, offline data and business information data from the data source by using kafka as the message queue; storing the business information data into a data warehouse.

3. The data analysis method according to claim 2, characterized in that, The step of performing real-time analysis and processing on the real-time data according to the stream processing technology specifically comprises the following steps: performing real-time analysis and processing on the real-time data according to a predefined rule and a complex event processing (CEP) library of Apache Flink technology to obtain real-time data after real-time analysis and processing; extracting the business information data from the data warehouse; associating and identifying the real-time data after real-time analysis and processing with the business information data to obtain associated and identified real-time data.

4. The data analysis method according to claim 3, characterized by, After the step of performing real-time analysis and processing on the real-time data according to the stream processing technology, and before the step of performing offline analysis and processing on the offline data according to the OLAP technology and the materialized view, the method further comprises the following step: storing the associated and identified real-time data into the data warehouse.

5. The data analysis method according to claim 4, characterized in that, The step of performing offline analysis and processing on the offline data according to the OLAP technology and the materialized view specifically comprises the following steps: extracting the associated and identified real-time data from the data warehouse; fusing the offline data and the associated and identified real-time data according to the OLAP technology to obtain a theme wide table under each business; aggregating and counting the theme wide table under each business according to the materialized view to obtain offline data after offline analysis and processing.

6. The data analysis method according to claim 5, characterized by, After the step of extracting the associated and identified real-time data from the data warehouse, and before the step of fusing the offline data and the associated and identified real-time data according to the OLAP technology to obtain the theme wide table under each business, the method further comprises the following step: performing data preprocessing on the offline data and the associated and identified real-time data, wherein the data preprocessing comprises but is not limited to normalization processing.

7. The data analysis method according to claim 5, characterized by, After the step of performing offline analysis and processing on the offline data according to the OLAP technology and the materialized view, the method further comprises the following step: scheduling a pre-designed computing resource according to a remote procedure call (RPC) technology to perform advanced data analysis on the offline data after offline analysis and processing, wherein the pre-designed computing resource comprises but is not limited to a computing script of a baseline model, and the advanced data analysis comprises but is not limited to model analysis.

8. An analysis processing apparatus of data, characterized by comprising: The method comprises the following steps: an accessing module, a real-time processing module and an offline processing module, the accessing module is configured to access raw data from a data source according to a message queue, wherein the raw data comprises real-time data and offline data, the real-time processing module is connected with the accessing module and is configured to perform real-time analysis and processing on the real-time data according to a stream processing technology, the offline processing module is connected with the accessing module and is configured to perform offline analysis and processing on the offline data according to an online analytical processing (OLAP) technology and a materialized view.

9. An electronic device, comprising: The computer program is stored in the memory and executed by the processor to implement the data analysis method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is stored in the memory and executed by the processor to implement the data analysis method according to any one of claims 1 to 7.

Images