Integrated processing method and device for batch data and streaming data, equipment and medium

Abstract

The invention relates to the technical field of data processing and the field of financial science and technology, and discloses an integrated processing method for batch data and streaming data, which comprises the following steps of: acquiring to-be-processed data from a data source, and configuring proper processing logic based on metadata information of the data, and applying the processing logic to a batch processing engine and a stream processing engine using an adaptation layer. Historical batch data is processed through a batch processing engine, real-time streaming data is processed through a streaming processing engine, and the historical batch data and the real-time streaming data generate respective processing results. And then, the two types of results are merged through an adaptation layer to generate a comprehensive data set, so that subsequent unified query and analysis are facilitated. According to the method and the system, integrated processing of batch data and streaming data is realized by establishing a unified framework. Through the adaptation layer, the data processing logic is respectively adapted to the batch processing engine and the stream processing engine for processing the data, so that the independent processing logic of two types of data through the same architecture is realized, and two sets of independent systems do not need to be maintained.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to a method, device, equipment and storage medium for integrated processing of batch data and streaming data. Background Art

[0002] In the financial industry, data analysis and processing have become an important basis for decision-making. Financial institutions need to process a large amount of historical and real-time transaction data in order to perform key tasks such as risk control, market analysis, and compliance management. There are two main ways to process data: batch computing and streaming computing. Batch computing is suitable for processing a large amount of accumulated historical data, while streaming computing is used to monitor and analyze newly generated data in real time.

[0003] Batch computing is a one-time processing of a set of data sets, usually executed uniformly after the data accumulates to a certain scale. This method is suitable for the analysis of historical data, but it cannot reflect the latest changes in the market or transactions in real time. In contrast, streaming computing can process and analyze data streams in real time, meeting the requirements of financial institutions for real-time risk assessment, real-time monitoring, and compliance management. However, the separation of batch computing and streaming computing has led to complexity in the design of data processing architecture in the financial industry.

[0004] In order to support both batch and streaming computing, the industry has introduced two architectural models:

[0005] Lambda architecture: Lambda architecture processes historical data through the batch processing layer, processes real-time data through the speed processing layer, and outputs the results to the service layer. The advantage of this architecture is that it can process historical data and support real-time computing, meeting the comprehensive needs of big data scenarios. However, its disadvantage is that two sets of logical systems need to be maintained, and the logic of the batch processing layer and the speed processing layer need to be consistent, which increases the complexity and maintenance cost of the system.

[0006] Kappa architecture: In order to solve the complexity of Lambda architecture, Kappa architecture simplifies the architecture, retains only the speed processing layer, and implements batch processing through streaming processing. Although this architecture simplifies system maintenance, it is not as flexible as Lambda architecture when processing large-scale historical data in the financial industry.

[0007] However, the financial industry has high requirements for data analysis and processing. The existing Lambda and Kappa architectures have the following shortcomings in the application process:

[0008] Although the Lambda architecture can support both batch and streaming computing, it requires the maintenance of two logical systems. It is very difficult to synchronize the batch processing layer and the speed processing layer, especially when processing financial data. Maintaining the consistency of the two systems is crucial, but it also increases the complexity of development and maintenance.

[0009] The Kappa architecture simplifies the system while implementing batch processing functions with stream processing as the core. However, for the historical data processing in the financial industry, the Kappa architecture is not as efficient and flexible as the Lambda architecture in batch data processing. In particular, there may be performance bottlenecks when processing large-scale historical data.

[0010] In financial scenarios, data accuracy and consistency are extremely important. The existing Lambda architecture may lead to inconsistent analysis results due to the logical inconsistency between the batch processing layer and the speed processing layer, which in turn affects the accuracy and timeliness of financial decisions.

[0011] In summary, the financial sector faces challenges in terms of system complexity, real-time performance, and data consistency in data processing architecture. Existing technologies are unable to efficiently support both batch and real-time data processing requirements. Summary of the invention

[0012] The main purpose of the present invention is to provide a method, device, equipment and storage medium for integrated processing of batch data and streaming data, aiming to solve the technical problem that the existing data processing technology relies on two independent processing systems, resulting in difficulty in coordinating the logic of batch computing and streaming computing, which increases the complexity of the system and the maintenance cost.

[0013] To achieve the above object, the present invention provides an integrated processing method for batch data and streaming data, comprising:

[0014] Acquire the data to be processed from the data source, wherein the data to be processed includes historical batch data and real-time stream data;

[0015] Based on the metadata information of the data to be processed, configure data processing logic;

[0016] Adapting the data processing logic to the batch processing engine and the stream processing engine through an adaptation layer;

[0017] Using the batch processing engine to process the historical batch data to generate corresponding batch data processing results;

[0018] Using the stream processing engine to process the real-time stream data to generate corresponding stream data processing results;

[0019] The batch data processing result and the streaming data processing result are merged by the adaptation layer to generate a comprehensive data set, which can be used for unified query.

[0020] Furthermore, to achieve the above-mentioned purpose, the present invention provides an integrated processing device for batch data and streaming data, comprising:

[0021] A data source connection module is used to obtain the data to be processed from the data source, wherein the data to be processed includes historical batch data and real-time stream data;

[0022] A processing logic configuration module, used to configure data processing logic based on metadata information of the data to be processed;

[0023] An adaptation layer module, used to adapt the data processing logic to the batch processing engine and the stream processing engine through the adaptation layer;

[0024] A batch processing engine module, used to use the batch processing engine to process the historical batch data and generate corresponding batch data processing results;

[0025] A stream processing engine module, used to process the real-time stream data using the stream processing engine to generate corresponding stream data processing results;

[0026] The data merging and querying module is used to merge the batch data processing results and the streaming data processing results through the adaptation layer to generate a comprehensive data set, and the comprehensive data set can be used for unified query.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an integrated processing program for batch data and streaming data stored in the memory and executable on the processor, wherein the integrated processing program for batch data and streaming data, when executed by the processor, implements the steps of the integrated processing method for batch data and streaming data as described above.

[0028] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored an integrated processing program for batch data and streaming data, and when the integrated processing program for batch data and streaming data is executed by a processor, the steps of the integrated processing method for batch data and streaming data as described above are implemented.

[0029] Beneficial effects: The present invention relates to the fields of data processing technology and financial technology, and discloses an integrated processing method for batch data and streaming data. By obtaining the data to be processed from the data source, a suitable processing logic is configured based on the metadata information of the data, and the processing logic is applied to the batch processing engine and the stream processing engine using the adaptation layer. The historical batch data is processed by the batch processing engine, and the real-time streaming data is processed by the stream processing engine, and the two generate their own processing results. Then, the two types of results are merged through the adaptation layer to generate a comprehensive data set, which is convenient for subsequent unified query and analysis. The present invention realizes the integrated processing of batch data and streaming data by establishing a unified framework. Through the adaptation layer, the data processing logic is adapted to the batch processing engine and the stream processing engine respectively, and the historical batch data is summarized and counted, and the real-time monitoring and anomaly detection are performed for the real-time streaming data. The independent processing logic of the two types of data is realized through the same architecture without the need to maintain two independent systems. Compared with the prior art, the system complexity is reduced, the maintenance process is simplified, and the consistency and coordination of the data processing results in batch and streaming computing scenarios are ensured. The merged comprehensive data set uniformly supports query, further improving the data processing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0031] Figure 1 A schematic diagram of an application environment of a method for integrated processing of batch data and streaming data in an embodiment of the present invention;

[0032] Figure 2 It is a flow chart of an embodiment of a method for integrated processing of batch data and streaming data of the present invention;

[0033] Figure 3 A schematic diagram of functional modules of a preferred embodiment of an integrated processing device for batch data and streaming data of the present invention;

[0034] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0035] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0037] The integrated processing method of batch data and streaming data provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the user end communicates with the server end through a network. The server end can obtain the data to be processed from the data source through the user end, configure appropriate processing logic based on the metadata information of the data, and use the adaptation layer to apply the processing logic to the batch processing engine and the stream processing engine. The historical batch data is processed by the batch processing engine, and the real-time stream data is processed by the stream processing engine, and the two generate their own processing results. Then, the two types of results are merged through the adaptation layer to generate a comprehensive data set, which is convenient for subsequent unified query and analysis. The present invention realizes the integrated processing of batch data and streaming data by establishing a unified framework. Through the adaptation layer, the data processing logic is adapted to the batch processing engine and the stream processing engine respectively, and the historical batch data is summarized and counted, and the real-time monitoring and anomaly detection are performed for the real-time stream data. The independent processing logic of the two types of data is realized through the same architecture without maintaining two independent systems. Compared with the prior art, the system complexity is reduced, the maintenance process is simplified, and the consistency and coordination of the data processing results in batch and stream computing scenarios are ensured. The merged comprehensive data set uniformly supports query, further improving the data processing efficiency and accuracy. The user end may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server end may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0038] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for integrated processing of batch data and streaming data provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0039] like Figure 2 As shown, the integrated processing method of batch data and streaming data proposed by the present invention includes the following steps:

[0040] S10, acquiring data to be processed from a data source, wherein the data to be processed includes historical batch data and real-time stream data;

[0041] In this embodiment, different data sources are identified and connected to obtain historical batch data and real-time streaming data respectively. The sources of batch data and streaming data may be different. Batch data is usually stored in a database or file system, while real-time streaming data is obtained through a message queue or a streaming processing platform.

[0042] In specific implementation, data can be pulled in batches from historical data sources through APIs or database connections, and real-time data streams can be monitored through messaging systems or stream processing frameworks (such as Kafka and Flink) to ensure continuous input of real-time data. The connection and acquisition process of the data source needs to be efficient and stable.

[0043] Historical batch data is a fixed data set that has been accumulated and is usually used for large-scale aggregation, statistics, and analysis. This data may come from transaction systems, log records, or archive databases.

[0044] Batch data acquisition is usually done by batch processing engines (such as Spark, Hadoop) and executed by time range or data conditions. For example, historical transaction data in a financial trading system can be exported in batches every day and transferred to an analysis system for processing.

[0045] Real-time streaming data is a continuously generated data stream that usually reflects the current state of the system or real-time events. The data needs to be continuously input in the form of a stream and processed instantly. This data is used for real-time monitoring, risk control, event detection, etc.

[0046] By listening to messages from real-time data sources through stream processing engines (such as Flink and Kafka Streams), data streams are received in real time and processed according to predefined time windows or event-driven mechanisms. In the financial field, this can be used to monitor market data changes in real time, such as stock trading, risk assessment, etc.

[0047] Example: In the financial data processing scenario, bank transaction records and market monitoring are two typical data sources. Banks can process historical transaction data in batches every day to analyze customers' transaction behaviors, account fund flows, etc. Through real-time streaming data, banks can monitor market changes in real time, conduct risk control and optimize decisions. By uniformly processing historical data and real-time data, banks can simultaneously obtain macro historical data analysis results and real-time market status feedback, improving decision-making efficiency and accuracy.

[0048] Through the above steps, the synchronous processing of historical batch data and real-time streaming data can be achieved, ensuring that historical data and real-time data can be processed simultaneously under a unified framework. This avoids the system complexity caused by the separate processing of historical data and real-time data, improves the efficiency and consistency of data processing, and is particularly suitable for the synchronous analysis needs of batch transaction data and real-time risk assessment data in the financial industry.

[0049] S20, configuring data processing logic based on the metadata information of the data to be processed;

[0050] In this embodiment, metadata information is a description of the structure, format, source, etc. of data, which is used to guide the configuration of data processing. Metadata information includes information such as data source, timestamp, data structure, field type, data format, etc. This information can provide a basis for subsequent data processing logic, ensuring that the processing logic can be adapted and executed for different types of data.

[0051] Extract metadata information from the data source, such as obtaining the structure and format of the data through database queries or file header information. For real-time streaming data, the timestamp and source of the data can be identified through the metadata of the streaming data message. The extracted metadata information is stored in the metadata repository for configuration calls of the data processing logic.

[0052] Based on the extracted metadata information, select and configure appropriate data processing logic. Historical batch data may require aggregation and statistical operations, while real-time streaming data may require monitoring and anomaly detection logic. The processing logic can include operations such as data cleaning, conversion, aggregation, and real-time event processing.

[0053] Select different processing logic based on metadata information. Through the rule engine or configuration management tool, dynamically generate SQL statements or processing scripts based on data type, source, timestamp and other information. For example, generate SQL statements for summary analysis of historical batch data, and configure real-time monitoring and alarm rules for real-time streaming data.

[0054] Example: In the financial sector, the transaction data processed by banks is divided into historical batch transaction data and real-time market monitoring data. Historical transaction data contains the customer's past transaction records and is suitable for summary and statistical analysis; while real-time market data is used to monitor market fluctuations and risk warnings. By configuring processing logic based on metadata information, banks can automatically select appropriate analysis models and logic to process different types of data. For example, historical data may be used to generate a customer's credit assessment report, while real-time data is used to detect potential risks brought about by market fluctuations, realizing the automation and intelligence of data processing.

[0055] By dynamically configuring data processing logic based on metadata information, it is possible to flexibly select appropriate processing logic according to the structure, format, and characteristics of different data sources. This improves the accuracy and flexibility of data processing and avoids the problem of reduced efficiency caused by unified data processing. It is particularly suitable for scenarios that require simultaneous processing of historical batch data and real-time streaming data, ensuring that both types of data can obtain the best processing results.

[0056] S30, adapting the data processing logic to the batch processing engine and the stream processing engine through an adaptation layer;

[0057] In this embodiment, the adaptation layer is an intermediate layer between the data processing logic and the computing engine (batch processing engine, stream processing engine), which is responsible for adjusting the configured data processing logic to a form suitable for each engine. The role of the adaptation layer is to convert different data processing logics into an execution format that conforms to the batch processing or stream processing engine.

[0058] The adaptation layer can use rule engines or script generation tools to translate high-level data processing logic into SQL queries and MapReduce tasks for batch processing engines, or real-time monitoring and aggregation tasks for stream processing engines. The adaptation layer can dynamically generate appropriate scripts or tasks based on the data type and processing requirements, and pass them to the batch processing engine or stream processing engine for execution.

[0059] The batch processing engine is used to process historical batch data, and the adaptation layer converts the data processing logic into batch processing tasks that the batch processing engine can understand. These tasks may include data cleaning, aggregation, sorting and other operations.

[0060] The adaptation layer generates SQL queries, Hadoop MapReduce tasks, or Spark jobs that conform to the batch engine specifications based on the characteristics of the batch data. These tasks can be executed in batches by time period or data volume, and the processing results may be summary statistics, data integration, etc. For example, in the financial field, the adaptation layer converts the processing logic of batch transaction data into SQL queries to calculate the total historical transaction amount of customers.

[0061] The stream processing engine is used to process real-time data. The adaptation layer converts the data processing logic into real-time computing tasks of the stream processing engine. These tasks may include real-time monitoring, anomaly detection, streaming aggregation, etc.

[0062] The adaptation layer adjusts the data processing logic to tasks that conform to the stream processing engine, such as real-time event processing, window computing, and other tasks. The adaptation layer generates stream processing scripts based on the timestamp or event trigger point of the data, which can process the data stream in real time. For example, in financial data stream processing, the adaptation layer adapts the market price fluctuation monitoring logic to Flink's window computing tasks for real-time monitoring of stock market changes.

[0063] Example description: In the financial field, the transaction data processed by banks includes both batch historical transaction data and real-time market monitoring data. The adaptation layer can convert the processing logic of historical transaction data into SQL queries in the batch processing engine according to specific data processing requirements, which is used to analyze the customer's historical transaction behavior. For real-time market data, the adaptation layer will generate real-time monitoring tasks and adapt them to the stream processing engine to monitor real-time market changes. This flexible adaptation method enables financial institutions to simultaneously cope with the needs of batch analysis of historical data and real-time processing of real-time data, greatly improving data processing efficiency and decision-making accuracy.

[0064] Through the design of the adaptation layer, different data processing logics can be automatically adapted to the batch processing engine and the stream processing engine under the same data processing framework. This avoids developers from manually writing processing logic for different engines, reducing system complexity and development costs. At the same time, the flexibility of the adaptation layer allows data processing logic to dynamically generate tasks. Whether it is batch processing or stream processing, it can be efficiently executed in the optimal computing environment, improving the overall performance and accuracy of data processing.

[0065] S40, using the batch processing engine to process the historical batch data to generate corresponding batch data processing results;

[0066] In this embodiment, the batch processing engine is used to perform one-time calculation and processing operations on a large amount of historical data. The historical batch data is usually accumulated data, which may include customer transaction records, log files, etc., which need to be processed on a large scale.

[0067] Batch processing engines (such as Apache Spark and Hadoop) read data in batches and perform predefined processing tasks. Processing tasks may include data cleaning (such as removing invalid records or duplicate data), data format conversion (unifying data format), and data aggregation (such as summarizing data by time period, calculating averages or sums, etc.). In the financial field, batch processing engines can process customer transaction data on a daily or monthly basis.

[0068] The processing result is the result data generated after summarizing, counting, and aggregating historical batch data. The batch processing result can be used for subsequent data analysis, report generation, or decision support.

[0069] After the batch processing engine completes the data processing, it stores the generated results in a database, distributed file system or data warehouse. The processing results may include customer transaction totals, account balance trends, data anomaly statistics, etc. The result data can be further queried or used by other systems. For example, a financial institution can generate a customer's historical transaction report or quarterly financial statements.

[0070] Example: In the financial sector, banks generate a large number of transaction records every day, which usually need to be processed in batches. The batch processing engine can process historical batch data every day to generate key data such as customer daily transaction summary and account fund flow. The processing results of these data can be used to analyze customer transaction behavior, generate financial statements or make risk control decisions. For example, banks can use batch processing engines to process customer transaction records every day, identify abnormal transactions and generate comprehensive reports.

[0071] The batch processing engine processes historical batch data, which can efficiently process large-scale historical data and ensure that batch data is uniformly and accurately summarized and analyzed. The processing process is automated, which reduces manual intervention and improves the efficiency and consistency of data processing. The results of batch processing provide reliable data support for subsequent data query, analysis, report generation and other applications.

[0072] S50, using the stream processing engine to process the real-time stream data to generate corresponding stream data processing results;

[0073] In this embodiment, the stream processing engine is used to perform real-time processing operations on continuously generated real-time data. Real-time stream data includes various real-time events, transaction data, sensor data, etc. The purpose of processing is to analyze, detect and respond to the data in real time.

[0074] Stream processing engines (such as Apache Flink and Kafka Streams) continuously receive data streams by listening to real-time data sources and process the data according to predefined processing logic. Common stream processing operations include real-time data cleaning, format conversion, window calculation, event monitoring, anomaly detection, etc. For example, in the financial field, stream processing engines can process stock trading data in real time, monitor market fluctuations, or detect trading anomalies.

[0075] The processing result is the result data generated by processing the real-time data through the stream processing engine. The stream data processing result can be real-time monitoring status, event triggering results, aggregated real-time data, etc., which are usually used for instant feedback and response.

[0076] The results of streaming data processing can be stored in a database or real-time monitoring system, or directly output to subsequent real-time processing modules. For example, the processing results may be monitoring indicators of real-time market data, abnormal transaction alerts, or real-time calculated financial indicators (such as volatility, transaction frequency, etc.). These results are usually used for financial decision-making, risk control, and real-time reporting.

[0077] Example: In the financial field, banks or financial institutions need to monitor market data and transaction status in real time. For example, a bank's trading system can use a stream processing engine to process real-time transaction data and market changes. The following are two typical application scenarios:

[0078] Stock market volatility monitoring: Real-time transaction data in the stock market is input into the system in the form of streams. The stream processing engine processes each transaction data in real time according to the set transaction threshold and market volatility rules, and calculates the volatility of stock prices. If the price fluctuation exceeds the set range, the stream processing engine immediately generates an early warning message and transmits the real-time monitoring results to traders or automated trading systems so that investment strategies can be quickly adjusted to avoid major risks.

[0079] Real-time transaction risk assessment: When banks conduct a large number of transactions, they need to identify suspicious financial transaction behaviors in real time. Through the stream processing engine, banks can analyze each real-time transaction. The system monitors the data in real time according to the set rules, such as transaction frequency, transaction amount, etc. If potential money laundering activities or abnormal transaction behaviors are detected, the stream processing engine generates a risk alert and sends the results to the bank's risk management system, which responds in real time to reduce risk exposure.

[0080] By using a stream processing engine, real-time streaming data can be processed and analyzed immediately, ensuring that data can be processed and fed back quickly after entering the system. Unlike a batch processing engine, a stream processing engine does not need to wait for data to accumulate before processing, but instead continuously receives and processes real-time data. This improves the timeliness and flexibility of data processing, and is particularly suitable for scenarios in the financial industry that require real-time monitoring and response, such as market volatility monitoring, transaction risk management, etc.

[0081] S60, merging the batch data processing result and the streaming data processing result through the adaptation layer to generate a comprehensive data set, and the comprehensive data set can be used for unified query.

[0082] In this embodiment, the adaptation layer is an intermediate layer for coordinating and integrating batch data processing results and streaming data processing results, and can process two different data types and standardize them. Batch data and streaming data usually have different processing cycles and structures. The role of the adaptation layer is to unify these processing results and ensure the consistency of data format and structure.

[0083] The adaptation layer can perform operations such as field mapping, format conversion, and timestamp alignment to standardize batch processing results and streaming data results. Then, through the merge operation, the summary results of historical data are combined with the monitoring results of real-time data to generate a unified comprehensive data set. The merge rules can be implemented based on timestamps, event triggers, or specific business logic.

[0084] A comprehensive data set is a unified data set formed by combining the processing results of batch data and real-time streaming data. This data set can contain historical summary data and real-time dynamic data, which can reflect past behavior patterns and current dynamic changes. A comprehensive data set includes not only the summary statistical results, but also the latest status of real-time data.

[0085] The merged data set is stored in a storage engine that supports OLAP (Online Analytical Processing) for efficient query and analysis. The generated comprehensive data set is indexed by time or other key dimensions for easy query and display. At the same time, data can be updated on demand to achieve continuous integration of batch data and streaming data.

[0086] The comprehensive data set is designed for unified query, that is, it can query historical data and real-time data at the same time. Users can retrieve summarized historical batch data and current streaming data at the same time in one query interface to meet various business needs.

[0087] Comprehensive data sets are stored in databases or data warehouses and queried through a unified SQL query interface or API. Queries can be based on time ranges, data types, or other conditions to enable combined analysis of batch and stream data. For example, in financial data, users can query historical transaction data for a certain period of time and current market dynamics data at the same time.

[0088] Example description: Banks need to perform end-of-day settlement of customer accounts every day, which includes processing a large amount of historical transaction data and real-time transaction data of the day. In traditional methods, banks usually process historical data first and then process real-time transactions of the day. The two are processed separately, which increases the complexity and time delay of data analysis. Through this embodiment, banks can process batch historical data and real-time data at the same time under a unified data framework to achieve real-time decision support.

[0089] The banking system first pulls the customer's transaction data for the past month (historical batch data) from the historical transaction database. The batch processing engine cleans, formats, and summarizes this data, such as calculating the total transaction amount for the past month, the weekly transaction frequency, and the summary of capital inflows and outflows.

[0090] At the same time, the banking system receives the real-time transaction data (real-time streaming data) of the day through the stream processing engine. The system processes the data at the moment the transaction occurs and monitors the transaction amount, transaction type, etc. in real time. The system can detect abnormal transaction behaviors in real time, such as abnormally high transfer amounts or excessively high transaction frequencies.

[0091] Through the adaptation layer, the system merges the historical transaction processing results of the past month with the real-time transaction data of the day. The adaptation layer aligns the timestamps and integrates the historical data with the real-time data into a comprehensive data set, summarizing the customer transaction behaviors of the day and historical data. This data set is stored in the bank's comprehensive data platform for real-time query by the finance and risk control departments.

[0092] Financial staff can access the comprehensive data set through a unified query interface to query all transactions of a customer in the past month, including the latest status of real-time transactions on the same day. The bank's risk control system can also query this comprehensive data set to conduct account health assessment and abnormal transaction detection.

[0093] By merging batch data processing results and streaming data processing results through the adaptation layer, seamless integration of historical data and real-time data is achieved. The generation of comprehensive data sets enables users to obtain a global view of historical data and the latest dynamics of real-time data in a unified query interface, simplifying the data query process and improving the efficiency and accuracy of data analysis, especially in scenarios that require reference to both historical trends and current conditions.

[0094] The present invention relates to the fields of data processing technology and financial technology, and discloses an integrated processing method for batch data and streaming data. By obtaining the data to be processed from the data source, a suitable processing logic is configured based on the metadata information of the data, and the processing logic is applied to the batch processing engine and the stream processing engine by using an adaptation layer. The historical batch data is processed by the batch processing engine, and the real-time streaming data is processed by the stream processing engine, and the two generate their own processing results. Then, the two types of results are merged through the adaptation layer to generate a comprehensive data set, which is convenient for subsequent unified query and analysis. The present invention realizes the integrated processing of batch data and streaming data by establishing a unified framework. Through the adaptation layer, the data processing logic is adapted to the batch processing engine and the stream processing engine respectively, and the historical batch data is summarized and counted, and the real-time monitoring and anomaly detection are performed for the real-time streaming data. The independent processing logic of the two types of data is realized through the same architecture without maintaining two independent systems. Compared with the prior art, the system complexity is reduced, the maintenance process is simplified, and the consistency and coordination of the data processing results in the batch and streaming computing scenarios are ensured. The merged comprehensive data set uniformly supports query, which further improves the data processing efficiency and accuracy.

[0095] In one embodiment, the above S20 includes:

[0096] S201, parsing the data to be processed and extracting metadata information, wherein the metadata information includes a data source identifier, a timestamp, a data structure, a field type, and a data format;

[0097] S202, based on the metadata information, selecting a data processing logic for aggregation and statistics for the historical batch data, and selecting a data processing logic for real-time monitoring and anomaly detection for the real-time stream data;

[0098] S203, configuring the selected data processing logic, generating an SQL statement or a processing engine script based on the metadata information, wherein the SQL statement or the processing engine script is used to perform data cleaning, format conversion and aggregation operations.

[0099] In this embodiment, the metadata information of the data to be processed describes the structure, source, time and other attributes of the data. By parsing the data, extracting these metadata helps to configure the processing logic later. The metadata information usually includes a data source identifier (indicating the data source system), a timestamp (indicating the data generation time), a data structure (defining data fields and formats), a field type (e.g., string, numeric type) and a data format (e.g., JSON, XML, etc.).

[0100] By parsing the header information or metadata file of the data to be processed, the metadata information can be automatically extracted. The system can obtain this information from the database, file header or metadata field in the stream data through the API, and store it in the metadata warehouse for subsequent processing.

[0101] For historical batch data, the processing logic mainly focuses on the aggregation, statistics, and analysis of past data. Based on the timestamp, data source, and other information in the metadata, you can select appropriate aggregation and statistics logic for historical data. This logic usually includes operations such as aggregation, grouping, calculating averages, and sums.

[0102] The system can automatically select aggregation and statistical processing logic based on the extracted metadata information, such as the time span or data type of the data. SQL statements or analysis engine scripts are generated to process historical batch data and output summary results, such as the total transaction amount by month or the historical account balance summary of customers.

[0103] For real-time streaming data, the processing logic focuses on real-time monitoring and event detection. By extracting the data source and timestamp information, the system can select real-time processing logic to detect abnormal behavior or key events in the data.

[0104] Based on metadata information, the system can dynamically generate stream processing engine scripts and configure real-time monitoring and alarm logic. For example, the system will analyze real-time transaction data, monitor abnormal large-amount transaction behaviors or market price fluctuations, generate real-time monitoring results and trigger alarm events.

[0105] After configuring the corresponding processing logic for different data types based on metadata information, the system will generate corresponding SQL statements or scripts to perform specific processing tasks. These SQL statements or scripts are responsible for performing operations such as data cleaning (such as removing invalid data), format conversion (unifying different formats into a processable standard format) and data aggregation.

[0106] The processing logic can automatically generate SQL queries to process historical data in batches. For streaming data, scripts for stream processing engines such as Flink or Kafka Streams can be generated to process real-time streaming data and perform real-time aggregation, filtering, and event detection operations.

[0107] This embodiment configures data processing logic based on metadata information to achieve dynamic processing for different data types (historical batch data and real-time streaming data), avoiding the limitations of fixed processing logic. It improves the accuracy and efficiency of data processing, and can quickly respond to the processing needs of different data sources, especially in the financial field, and can flexibly respond to the needs of transaction data aggregation and real-time market monitoring in different time periods.

[0108] In one embodiment, in the above S60, merging the batch data processing result and the streaming data processing result by the adaptation layer to generate a comprehensive data set includes:

[0109] S601, standardizing batch data processing results and streaming data processing results through the adaptation layer to unify field types, data structures and timestamp formats;

[0110] S602, merging the standardized batch data processing results and the streaming data processing results to generate a comprehensive data set;

[0111] S603, importing the integrated data set into a storage engine supporting online processing, where the storage engine is used to manage and query the combined data set;

[0112] S604, organizing and managing the comprehensive data set through the index or alias mechanism of the storage engine, so that batch data processing results and streaming data processing results can be uniformly queried and accessed in the same data set.

[0113] In this embodiment, batch data and streaming data may differ in format, field type, timestamp, etc. The task of the adaptation layer is to standardize these differences to ensure that the data can be merged under the same framework. Standardization mainly involves field types (such as unification of integers, strings, etc.), data structures (such as adjustment of nested structures or flat structures), and timestamp formats (ensuring seamless connection of time series data).

[0114] The adaptation layer converts the field types of batch data and streaming data into a consistent format based on the metadata information of the two. Timestamps are standardized using a unified time standard (such as ISO 8601 format) so that batch and real-time data can be sorted and compared in the same time unit.

[0115] The core task of this solution is to merge batch data and streaming data. The merge operation combines the processing results of historical data with the latest status of real-time data to generate a comprehensive data set that contains both historical information and real-time dynamics.

[0116] The system merges by timestamp or other key fields through the adaptation layer. For example, batch data and real-time data are aligned by timestamp to ensure the integrity of time series data. The merged data set retains the summary information of historical data while reflecting the latest changes in real-time data.

[0117] After the comprehensive data set is generated, it needs to be stored and provide subsequent query and processing services. The storage engine for online processing (OLAP engine) supports fast query and analysis of large-scale data. The storage engine is responsible for managing the combined data set and providing optimized access paths based on query requirements.

[0118] Store comprehensive data sets in databases or data warehouses that support OLAP, such as Elasticsearch, ClickHouse, etc. These storage engines provide efficient indexing and data compression technologies, supporting real-time query and analysis of large-scale data.

[0119] In order to efficiently manage and access comprehensive data sets, the storage engine uses indexes and alias mechanisms to organize and manage data. Indexes speed up data access by optimizing query paths, and alias mechanisms allow users to query data by unified names without having to worry about the specific storage location or format behind the data.

[0120] The system can create multi-dimensional indexes for comprehensive data sets (such as by time, user, transaction type, etc.) to quickly respond to different query requirements. The alias mechanism is used to simplify the query interface and provide a consistent user experience.

[0121] This embodiment uses the adaptation layer to standardize batch data and streaming data and then merge them. The generated comprehensive data set can effectively support unified query. The adaptation layer solves the problem of inconsistent formats and timestamps between historical data and real-time data, ensuring that the two types of data can be seamlessly integrated. Through the online processing engine, users can simultaneously obtain summary information of historical data and the latest developments of real-time data in the same query interface, greatly improving the efficiency of data query and analysis and simplifying data management and access processes.

[0122] In one embodiment, the above S40 includes:

[0123] S401, importing the historical batch data into a batch processing engine, and performing data cleaning, format conversion and preliminary data aggregation operations through the batch processing engine in combination with a data processing module;

[0124] S402, performing corresponding business processing and data filtering operations on the preliminarily processed historical batch data through a custom logic module;

[0125] S403, generating and outputting batch data processing results of historical batch data, wherein the batch data processing results include batch data summary information.

[0126] In this embodiment, the batch processing engine is responsible for processing large-scale historical batch data. Before data processing, data cleaning, format conversion and preliminary aggregation are necessary pre-processing steps. Data cleaning is used to remove invalid or duplicate records, format conversion ensures the uniformity of data structure, and preliminary aggregation performs basic statistical and summary processing on the data.

[0127] The system loads historical batch data through a batch processing engine (such as Spark or Hadoop), and first performs data cleaning operations to remove invalid data or correct erroneous data. Then, the system converts the data format uniformly, such as standardizing different data types into a unified format. Finally, the batch processing engine performs preliminary aggregation operations to perform data aggregation by time period or other business logic, such as calculating the total transaction amount or customer activity summary for each month.

[0128] The custom logic module allows for more in-depth processing of historical batch data based on specific business needs. The tasks of this module include complex business calculations and data filtering operations, which can be customized according to the specific needs of an enterprise or industry.

[0129] Custom logic modules support user-defined processing logic, such as classifying data according to specific business rules, calculating complex indicators, or filtering out data that does not meet the conditions. Specific implementations in the financial field may include filtering data based on customer type or transaction risk level, or calculating specific business indicators (such as customer credit scores).

[0130] Finally, the historical batch data processing results after cleaning, format conversion, aggregation and business processing are generated and output. The results usually include summary information of batch data, such as the total amount, average value or other summary statistics of transaction data.

[0131] The batch data processing results can be stored in a database or data warehouse for subsequent query and analysis. The system can output these results as reports, statistical tables or further data sets for decision support or business optimization. For example, a financial institution can generate a summary report of a customer's historical transactions, showing account balance trends or large transaction analysis results.

[0132] This embodiment uses a batch processing engine to perform multi-step processing on historical batch data, including data cleaning, format conversion, aggregation and other operations, and then implements flexible business processing through a custom logic module. The final batch data processing result not only contains summary information, but also combines custom processing of business logic. It not only improves the efficiency of data processing, but also improves the flexibility of data analysis, and is particularly suitable for scenarios that require in-depth analysis of large-scale historical data, such as transaction data aggregation and customer behavior analysis in the financial field.

[0133] In one embodiment, the above S50 includes:

[0134] S501, preprocessing the received real-time stream data through the stream processing engine, wherein the preprocessing includes data cleaning, format conversion and timestamp alignment operations;

[0135] S502, performing real-time processing on the real-time stream data through a custom logic module, wherein the real-time processing includes real-time monitoring, anomaly detection, and event stream processing;

[0136] S503: Generate and output a streaming data processing result, where the streaming data processing result includes aggregation information of the real-time streaming data.

[0137] In this embodiment, the data received by the stream processing engine is real-time data, and preprocessing operations are required before processing, including data cleaning, format conversion, and timestamp alignment. Data cleaning is used to filter out invalid, duplicate, or incomplete data, format conversion ensures that the data is in a unified format, and timestamp alignment is used to synchronously process data generated from different sources or at different times.

[0138] The system processes real-time streaming data through a stream processing engine (such as Apache Flink, Kafka Streams), first cleaning up invalid information such as noise data and duplicate data. Then, the stream processing engine converts the data into the required standard format, such as converting the JSON format into a structured data table format. Timestamp alignment adjusts the time field of the data so that events from multiple data sources can be processed based on a unified time base.

[0139] The custom logic module in the real-time processing stage can be used to define specific business rules to monitor data flows in real time, detect anomalies, and handle specific events. This module allows you to customize the data processing logic according to different business scenarios.

[0140] The system uses custom logic modules to monitor real-time data in real time and detect abnormal data points or transactions. Real-time monitoring can be monitoring changes in data, such as monitoring fluctuations in transaction frequency or transaction amount. Anomaly detection logic is used to discover data behaviors that do not meet expectations, such as suspicious transactions or abnormal changes in market prices. Event stream processing allows further actions to be executed based on event-based triggering rules, such as sending alarms in real time or triggering automatic processing mechanisms.

[0141] The result of streaming data processing is the aggregated information of data after real-time processing, usually including statistics and monitoring indicators of real-time data. The result not only includes the processing result of a single data, but also the aggregated information after summary and statistics.

[0142] The system generates results from the data after real-time processing, such as calculating the total transaction volume in the current time window, the price fluctuation range, or the alarm status in the monitoring system. The generated streaming data processing results can be stored in the real-time database or directly output to the real-time monitoring system for users or other systems to query and analyze.

[0143] This embodiment uses a stream processing engine and a custom logic module to achieve rapid preprocessing and business logic processing of real-time stream data. The real-time processing results not only support real-time monitoring, but also dynamically detect abnormal situations and generate aggregate information, which improves the timeliness and accuracy of data processing. The system can flexibly refine real-time data according to business needs, which is particularly suitable for data processing scenarios that require instant response, such as transaction monitoring and market analysis in the financial field.

[0144] In one embodiment, the above S10 includes:

[0145] S101, identifying and connecting a data source, where the data source includes a historical batch data source and a real-time stream data source;

[0146] S102, performing a data pulling operation according to a predetermined time period or data condition, and using a batch processing engine to read historical batch data in batches from the historical batch data source;

[0147] S103, establishing a monitoring mechanism for the real-time data source, receiving real-time stream data from the real-time stream data source in an event-driven manner, and inputting the real-time stream data into the stream processing engine.

[0148] In this embodiment, the system needs to clarify which data sources are historical batch data sources and which are real-time streaming data sources. These data sources may be databases, file storage systems, or streaming data platforms.

[0149] The system connects to the database or streaming data platform through the API to identify historical batch data sources and real-time streaming data sources. For batch data sources, the system usually connects to SQL databases or file systems such as HDFS, while real-time data sources may be stream processing platforms such as Kafka and Flink. After the connection is successful, the system marks these data sources as available and prepares for data reading.

[0150] Batch data reading is usually performed based on a set time period or conditions (such as transaction date, data update frequency). The batch processing engine is responsible for reading historical batch data under specified conditions from the data source in batches.

[0151] Query operations are performed on historical batch data sources through batch processing engines (such as Hadoop and Spark), and data is read according to the set time period or data conditions. For example, in a banking system, a batch processing engine can read transaction records by day or month. The system imports this data into the data processing module in batches for cleaning, format conversion, and other operations.

[0152] For real-time streaming data, the system needs to continuously receive the real-time data stream through a monitoring mechanism. The monitoring mechanism can detect new data in the data source and process it in real time in an event-driven manner.

[0153] The system establishes a connection with a streaming data source (such as Kafka or Flink) and sets up event monitoring. When new data arrives at the real-time streaming data source, the system automatically receives it and inputs the data into the stream processing engine for processing. The event-driven approach ensures that the system can respond to real-time data changes and process them in a timely manner. For example, in a stock trading system, when new trading data arrives, the system receives it in real time and performs analysis operations.

[0154] Through the above steps, this embodiment can process historical batch data and real-time streaming data at the same time, ensuring that batch data and real-time data are efficiently processed under the same framework. The batch processing engine can flexibly obtain historical data according to time periods or data conditions, while the stream processing engine can receive and process streaming data in real time to ensure the timeliness and continuity of the data. This method effectively improves the flexibility and response speed of data processing, and is particularly suitable for transaction analysis and risk monitoring in the financial industry.

[0155] In one embodiment, in the above S60, after generating the comprehensive data set, the following step is further included:

[0156] S605, receiving a query request, wherein the query request includes a query condition that limits a time range or a data type;

[0157] S606, performing a query operation in the comprehensive data set based on the query condition, and respectively retrieving batch data processing results and streaming data processing results;

[0158] S607, extracting summary data or real-time data that meets the query condition from the comprehensive data set, and returning the query result.

[0159] In this embodiment, the query request includes query conditions specified by the user or the system, which usually include a time range (such as transaction data within a certain time period) or a specific data type (such as customer transaction records, market data, etc.). These conditions are used to filter and limit the scope of query results to ensure the accuracy and efficiency of the query.

[0160] The system receives query requests initiated by users or the system through the query interface. The request can contain multiple query conditions, such as a specific time period (such as transaction data in the past week) or data type (such as transaction data, account data). These query conditions will be passed to the subsequent query processing module for data filtering.

[0161] After receiving the query request, the system executes the query operation in the comprehensive data set. The comprehensive data set is composed of batch data and streaming data, so the query needs to retrieve the processing results of batch data and real-time streaming data respectively.

[0162] Based on the conditions provided by the user, the query engine first searches for historical data that meets the conditions in the batch data processing results, and then searches for the corresponding real-time data in the streaming data processing results. For example, if the query condition is "trading data for the past week", the system will extract historical transaction records from the batch data and real-time transaction records from the streaming data. The query can be implemented through the SQL or NoSQL query interface.

[0163] The system will extract qualified data from the comprehensive data set, including batch data summary information or real-time streaming data. The query results can be statistical results of historical data or the latest status of current real-time data. Finally, the query results are packaged and returned to the user or system.

[0164] After the system extracts data through the query engine, it performs data aggregation or real-time monitoring operations to generate comprehensive query results. For example, a bank's query system can return the total transaction amount of the past week and the latest status of real-time transactions on the day. The results can be returned to the user in the form of structured data, reports or graphical displays.

[0165] This embodiment realizes the ability to retrieve historical batch data and real-time streaming data at the same time by receiving query requests and executing queries in the comprehensive data set. Users can quickly obtain summary data and the latest real-time data in the comprehensive data set through a single query interface, greatly improving the efficiency and accuracy of data query.

[0166] In one embodiment, a device for integrated processing of batch data and streaming data is provided, and the device for integrated processing of batch data and streaming data corresponds one-to-one to the method for integrated processing of batch data and streaming data in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the integrated processing device for batch data and streaming data of the present invention. Data source connection module 10, processing logic configuration module 20, adaptation layer module 30, batch processing engine module 40, stream processing engine module 50 and data merging and query module 60. The functional modules are described in detail as follows:

[0167] The data source connection module 10 is used to obtain the data to be processed from the data source, and the data to be processed includes historical batch data and real-time stream data;

[0168] A processing logic configuration module 20, configured to configure data processing logic based on metadata information of the data to be processed;

[0169] An adaptation layer module 30, used to adapt the data processing logic to the batch processing engine and the stream processing engine through the adaptation layer;

[0170] A batch processing engine module 40, configured to process the historical batch data using the batch processing engine to generate corresponding batch data processing results;

[0171] The stream processing engine module 50 is used to process the real-time stream data using the stream processing engine to generate corresponding stream data processing results;

[0172] The data merging and querying module 60 is used to merge the batch data processing results and the streaming data processing results through the adaptation layer to generate a comprehensive data set, and the comprehensive data set can be used for unified query.

[0173] In one embodiment, the processing logic configuration module 20 is specifically configured to:

[0174] Parsing the data to be processed and extracting metadata information, wherein the metadata information includes data source identification, timestamp, data structure, field type and data format;

[0175] Based on the metadata information, select data processing logic for aggregation and statistics for the historical batch data, and select data processing logic for real-time monitoring and anomaly detection for the real-time stream data;

[0176] The selected data processing logic is configured to generate SQL statements or processing engine scripts based on the metadata information, wherein the SQL statements or processing engine scripts are used to perform data cleaning, format conversion, and aggregation operations.

[0177] In one embodiment, the data merging and querying module 60 is specifically used to:

[0178] The batch data processing results and the streaming data processing results are standardized by the adaptation layer to unify the field type, data structure and timestamp format;

[0179] Merge the standardized batch data processing results and streaming data processing results to generate a comprehensive data set;

[0180] Importing the integrated data set into a storage engine supporting online processing, wherein the storage engine is used to manage and query the combined data set;

[0181] The comprehensive data set is organized and managed through the index or alias mechanism of the storage engine, so that batch data processing results and streaming data processing results can be uniformly queried and accessed in the same data set.

[0182] In one embodiment, the batch processing engine module 40 is specifically configured to:

[0183] Importing the historical batch data into a batch processing engine, and performing data cleaning, format conversion and preliminary data aggregation operations through the batch processing engine in combination with a data processing module;

[0184] Perform corresponding business processing and data filtering operations on the preliminarily processed historical batch data through custom logic modules;

[0185] A batch data processing result of the historical batch data is generated and output, wherein the batch data processing result includes batch data summary information.

[0186] In one embodiment, the stream processing engine module 50 is specifically configured to:

[0187] Preprocessing the received real-time stream data by the stream processing engine, wherein the preprocessing includes data cleaning, format conversion and timestamp alignment operations;

[0188] Perform real-time processing on real-time streaming data through a custom logic module, wherein the real-time processing includes real-time monitoring, anomaly detection, and event stream processing;

[0189] Generate and output a streaming data processing result, wherein the streaming data processing result includes aggregated information of real-time streaming data.

[0190] In one embodiment, the data source connection module 10 is specifically used for:

[0191] Identify and connect data sources, including historical batch data sources and real-time streaming data sources;

[0192] Performing a data pull operation according to a predetermined time period or data condition, and using a batch processing engine to read historical batch data in batches from the historical batch data source;

[0193] A monitoring mechanism for a real-time data source is established, and real-time streaming data is received from the real-time streaming data source in an event-driven manner, and input into a stream processing engine.

[0194] In one embodiment, the data merging and querying module 60 is specifically used to:

[0195] receiving a query request, wherein the query request includes a query condition that limits a time range or a data type;

[0196] Performing a query operation in the comprehensive data set based on the query condition to retrieve batch data processing results and streaming data processing results respectively;

[0197] Extract summary data or real-time data that meets the query conditions from the comprehensive data set and return the query results.

[0198] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a service-side method for integrated processing of batch data and streaming data.

[0199] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a method for integrated processing of batch data and streaming data

[0200] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0201] Acquire the data to be processed from the data source, wherein the data to be processed includes historical batch data and real-time stream data;

[0202] Based on the metadata information of the data to be processed, configure data processing logic;

[0203] Adapting the data processing logic to the batch processing engine and the stream processing engine through an adaptation layer;

[0204] Using the batch processing engine to process the historical batch data to generate corresponding batch data processing results;

[0205] Using the stream processing engine to process the real-time stream data to generate corresponding stream data processing results;

[0206] The batch data processing result and the streaming data processing result are merged by the adaptation layer to generate a comprehensive data set, which can be used for unified query.

[0207] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0208] Acquire the data to be processed from the data source, wherein the data to be processed includes historical batch data and real-time stream data;

[0209] Based on the metadata information of the data to be processed, configure data processing logic;

[0210] Adapting the data processing logic to the batch processing engine and the stream processing engine through an adaptation layer;

[0211] Using the batch processing engine to process the historical batch data to generate corresponding batch data processing results;

[0212] Using the stream processing engine to process the real-time stream data to generate corresponding stream data processing results;

[0213] The batch data processing result and the streaming data processing result are merged by the adaptation layer to generate a comprehensive data set, which can be used for unified query.

[0214] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0215] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0216] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0217] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for integrated processing of batch data and streaming data, characterized in that: The following steps are involved: Acquire the data to be processed from the data source, wherein the data to be processed includes historical batch data and real-time stream data; Based on the metadata information of the data to be processed, configure data processing logic; Adapting the data processing logic to the batch processing engine and the stream processing engine through an adaptation layer; Using the batch processing engine to process the historical batch data to generate corresponding batch data processing results; Using the stream processing engine to process the real-time stream data to generate corresponding stream data processing results; The batch data processing result and the streaming data processing result are merged by the adaptation layer to generate a comprehensive data set, which can be used for unified query.

2. The integrated processing method of batch data and streaming data according to claim 1, characterized in that: Based on the metadata information of the data to be processed, configuring data processing logic includes: Parsing the data to be processed and extracting metadata information, wherein the metadata information includes data source identification, timestamp, data structure, field type and data format; Based on the metadata information, select data processing logic for aggregation and statistics for the historical batch data, and select data processing logic for real-time monitoring and anomaly detection for the real-time stream data; The selected data processing logic is configured to generate SQL statements or processing engine scripts based on the metadata information, wherein the SQL statements or processing engine scripts are used to perform data cleaning, format conversion, and aggregation operations.

3. The integrated processing method of batch data and streaming data according to claim 1, characterized in that: The batch data processing result and the streaming data processing result are combined by the adaptation layer to generate a comprehensive data set, including: The batch data processing results and the streaming data processing results are standardized by the adaptation layer to unify the field type, data structure and timestamp format; Merge the standardized batch data processing results and streaming data processing results to generate a comprehensive data set; Importing the integrated data set into a storage engine supporting online processing, wherein the storage engine is used to manage and query the combined data set; The comprehensive data set is organized and managed through the index or alias mechanism of the storage engine, so that batch data processing results and streaming data processing results can be uniformly queried and accessed in the same data set.

4. The integrated processing method of batch data and streaming data according to claim 1, characterized in that: Using the batch processing engine to process the historical batch data to generate corresponding batch data processing results includes: Importing the historical batch data into a batch processing engine, and performing data cleaning, format conversion and preliminary data aggregation operations through the batch processing engine in combination with a data processing module; Perform corresponding business processing and data filtering operations on the preliminarily processed historical batch data through custom logic modules; A batch data processing result of the historical batch data is generated and output, wherein the batch data processing result includes batch data summary information.

5. The integrated processing method of batch data and streaming data according to claim 1, characterized in that: Using the stream processing engine to process the real-time stream data to generate corresponding stream data processing results includes: Preprocessing the received real-time stream data by the stream processing engine, wherein the preprocessing includes data cleaning, format conversion and timestamp alignment operations; Perform real-time processing on real-time streaming data through a custom logic module, wherein the real-time processing includes real-time monitoring, anomaly detection, and event stream processing; Generate and output a streaming data processing result, wherein the streaming data processing result includes aggregated information of real-time streaming data.

6. The integrated processing method of batch data and streaming data according to claim 1, characterized in that: Obtaining data to be processed from a data source, wherein the data to be processed includes historical batch data and real-time stream data, including: Identify and connect data sources, including historical batch data sources and real-time streaming data sources; Performing a data pull operation according to a predetermined time period or data condition, and using a batch processing engine to read historical batch data in batches from the historical batch data source; A monitoring mechanism for a real-time data source is established, and real-time streaming data is received from the real-time streaming data source in an event-driven manner, and input into a stream processing engine.

7. The integrated processing method of batch data and streaming data as claimed in claim 1, characterized in that: After generating the comprehensive dataset, it also includes: receiving a query request, wherein the query request includes a query condition that limits a time range or a data type; Performing a query operation in the comprehensive data set based on the query condition to retrieve batch data processing results and streaming data processing results respectively; Extract summary data or real-time data that meets the query conditions from the comprehensive data set and return the query results.

8. An integrated processing device for batch data and streaming data, characterized in that: The integrated processing device for batch data and streaming data includes: A data source connection module is used to obtain the data to be processed from the data source, wherein the data to be processed includes historical batch data and real-time stream data; A processing logic configuration module, used to configure data processing logic based on metadata information of the data to be processed; An adaptation layer module, used to adapt the data processing logic to the batch processing engine and the stream processing engine through the adaptation layer; A batch processing engine module, used to use the batch processing engine to process the historical batch data and generate corresponding batch data processing results; A stream processing engine module, used to process the real-time stream data using the stream processing engine to generate corresponding stream data processing results; The data merging and querying module is used to merge the batch data processing results and the streaming data processing results through the adaptation layer to generate a comprehensive data set, and the comprehensive data set can be used for unified query.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an integrated processing program for batch data and streaming data stored in the memory and executable on the processor. When the integrated processing program for batch data and streaming data is executed by the processor, the steps of the integrated processing method for batch data and streaming data described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a batch data and streaming data integrated processing program, which, when executed by a processor, implements the steps of the batch data and streaming data integrated processing method as described in any one of claims 1 to 7.

Images