Data processing method, device and system, storage medium and product

Abstract

The invention discloses a data processing method, device and system, a storage medium and a product, and relates to the technical field of data processing.The method comprises the steps that source data flowing in in real time is obtained; carrying out integrated real-time calculation processing on the source data through a real-time calculation engine; and providing the processed real-time streaming data to a downstream system. According to the method, one set of development tool and framework can be used for processing data, the complexity of development and operation is reduced, the risk of data inconsistency is reduced, the accuracy and reliability of the data are improved, through a batch stream data fusion scheme, the delay of data from generation to analysis is reduced, rapid decision making is supported, and the competitiveness is improved. Besides, according to the technical scheme of the invention, batch calculation processing is introduced as a bottom link, so that the data quality is further guaranteed, and higher requirements on real-time performance, efficiency, consistency and data quality in a big data scene are met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, system, storage medium and product. Background Art

[0002] Batch data processing is the most common method in traditional data warehouses. It is usually performed in a specific time period (such as daily or weekly) to process a large amount of historical data. The general process of batch data processing is: data is first extracted from various source systems (such as transaction systems), then cleaned, converted, integrated, and finally loaded into the data warehouse. The advantage of batch processing is that it has strong processing capabilities and can process large-scale data, but the disadvantage is high latency.

[0003] With the development of stream processing technology, real-time data processing has become increasingly important. It can process and analyze a continuous stream of real-time data streams, such as user behavior data, and process and respond to data almost instantly when it is generated. It is suitable for scenarios that require real-time insights and quick decision-making.

[0004] Traditional data processing architecture tends to process batch data and real-time streaming data separately. This approach often leads to data consistency issues, while increasing the complexity and cost of development and operations. At the same time, the problem of poor timeliness of batch data has not been improved. Summary of the invention

[0005] The main purpose of this application is to provide a data processing method, device, system, storage medium and product, aiming to effectively solve the timeliness problem of batch data, reduce the risk of data inconsistency, and reduce the complexity and cost of development and operation and maintenance.

[0006] To achieve the above objectives, the present application proposes a data processing method, which includes:

[0007] Get real-time incoming source data;

[0008] Performing integrated real-time computing and processing on the source data through a real-time computing engine;

[0009] The processed real-time streaming data is provided to the downstream system through the file system.

[0010] In one embodiment, after the step of performing integrated real-time computing processing on the source data by the real-time computing engine, the method further includes:

[0011] The processed real-time stream data is written back to Kafka in real time for real-time data storage.

[0012] In one embodiment, the step of providing the processed real-time streaming data to a downstream system via a file system includes:

[0013] Provide real-time streaming data stored in Kafka to real-time downstream systems and / or offline downstream systems through the file system.

[0014] In one embodiment, before the step of providing the real-time stream data stored in Kafka to the offline downstream system through the file system, the method further includes:

[0015] The processed real-time streaming data is verified for data quality through a preset data quality verification mechanism;

[0016] If the verification is passed, the offline data that has passed the verification will be stored in the offline data warehouse, and the offline stored data will be provided to the offline downstream system.

[0017] In one embodiment, after the step of performing data quality verification on the processed real-time stream data through a preset data quality verification mechanism, the method further includes:

[0018] If the test fails, the preset backup link is automatically triggered to process the source data.

[0019] In one embodiment, if the inspection fails, the step of automatically triggering a preset backup link to process the source data includes:

[0020] If the test fails, the preset backup link is automatically triggered to perform batch calculations on the source data through a batch processing engine, and the batch calculated data is stored in an offline data warehouse and provided to an offline downstream system.

[0021] The present application also proposes a data processing device, the device comprising:

[0022] The acquisition module is used to obtain the real-time incoming source data;

[0023] A computing module, used for performing integrated real-time computing processing on the source data through a real-time computing engine;

[0024] The transmission module is used to provide the processed real-time streaming data to the downstream system through the file system.

[0025] The present application also proposes a data processing system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0026] The present application also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0027] The present application also proposes a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0028] One or more technical solutions proposed in this application have at least the following technical effects:

[0029] Provided are a data processing method, device, system, storage medium and product, which obtains real-time incoming source data; performs integrated real-time computing processing on the source data through a real-time computing engine; and provides the processed real-time streaming data to the downstream system through a file system. The technical solution of the present application uses a real-time computing engine to perform integrated real-time computing processing on the source data, so that a set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, the real-time computing engine is used to perform integrated real-time computing processing on the source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. In addition, the present technical solution introduces batch computing processing as a bottom-up link to further ensure data quality. Therefore, the technical solution of the present application can meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0033] Figure 2 A flowchart of the second embodiment of the data processing method of the present application is provided;

[0034] Figure 3 A flowchart of the third embodiment of the data processing method of the present application is provided;

[0035] Figure 4 A complete flowchart of the data processing method provided in the embodiment of the present application;

[0036] Figure 5 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0037] Figure 6 A schematic diagram of the system structure of the hardware operating environment involved in the data processing method of the embodiment of the present application.

[0038] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0040] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0041] The main solution of the embodiment of the present application is: by obtaining the real-time inflow source data; performing integrated real-time computing processing on the source data through the real-time computing engine; and providing the processed real-time streaming data to the downstream system through the file system. The technical solution of the present application uses a real-time computing engine to perform integrated real-time computing processing on the source data, so that a set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, the real-time computing engine is used to perform integrated real-time computing processing on the source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. In addition, the technical solution of the present application introduces batch computing processing as a bottom-up link to further ensure data quality. Therefore, the technical solution of the present application can meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios.

[0042] Traditional data processing architectures often process batch data and real-time streaming data independently, which often leads to data consistency problems and significantly increases the complexity and cost of development and operation. In addition, this approach does not effectively solve the timeliness problem of batch data.

[0043] Therefore, in order to cope with the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios, the embodiments of the present application provide a batch-stream fusion data solution to reduce the delay from data generation to analysis, support rapid decision-making, and enhance competitiveness. Use a set of development tools and frameworks to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve data accuracy and reliability. In addition, this technical solution introduces batch computing processing as a bottom-up link to further ensure data quality.

[0044] The present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data processing method of the present application.

[0045] In this embodiment, the data processing method includes steps S10 to S30, each of which is explained in detail below.

[0046] like Figure 1 As shown, the first embodiment of the present application proposes a data processing method, the method comprising:

[0047] Step S10, obtaining real-time inflow source data;

[0048] The technical solution of this embodiment can use a set of development tools and frameworks to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, improve data accuracy and reliability, and reduce the delay from data generation to analysis through batch and stream fusion data solutions, support rapid decision-making, and enhance competitiveness.

[0049] The execution subject of this embodiment may be a data processing device, which may be carried on a terminal or a server in the form of hardware or software.

[0050] As an implementation mode, the data processing device may obtain the real-time inflowing source data after receiving the data processing instruction.

[0051] Source data refers to the original data that has not been processed or converted. It is the starting point of all data processing. There are many types of source data, which can be classified according to different standards such as its source, format, and purpose.

[0052] Common source data types include: database data, file data, online data, transaction data, survey and report data, etc.

[0053] The database data may include: table data in relational databases such as MySQL, Oracle, and SQL Server, and documents and key-value pairs in non-relational databases such as MongoDB, Cassandra, and Redis.

[0054] Online data includes data on websites and social media, such as web page content, user behavior data (click stream, browsing history), etc.

[0055] Transaction data includes shopping records, such as purchase history and payment information on e-commerce platforms, and banking transactions, such as deposit and withdrawal records and transfer records of bank accounts.

[0056] The data stream in this embodiment may originate from various source systems, such as a transaction system.

[0057] In the financial field, source data refers to the original data generated, collected or obtained by financial institutions during their operations. These data are the basis for financial analysis and decision-making. Among them, common types of source data in the financial field include: transaction data, customer data, market data, financial statement data, credit data, risk data, and payment data.

[0058] Transaction data includes: stock trading data, such as buying and selling prices, trading volumes, trading times, etc.; fixed income trading data, such as bond buying and selling information, yields, etc.; foreign exchange trading data, such as exchange rates, trading volumes, trading times, etc.; commodity trading data, such as commodity prices, trading volumes, delivery dates, etc.

[0059] Customer data includes: personal information, such as name, address, contact information, ID number, etc.; financial information, such as income, assets, liabilities, credit score, etc.; transaction behavior, such as customer's transaction records, investment preferences, transaction frequency, etc.

[0060] Payment data includes: electronic payment data, such as transaction information of online payment and mobile payment; bank transfer data, such as detailed information on fund transfers between accounts, etc.

[0061] Source data in the financial field usually needs to be strictly processed and analyzed to meet regulatory requirements, risk management, customer service and business decision-making needs. Financial institutions usually use advanced data analysis tools and technologies to extract valuable information from these source data.

[0062] This embodiment uses a real-time computing engine to perform integrated real-time computing processing on source data. The real-time computing engine is used to process and analyze continuously flowing data without storing the data for subsequent processing. Real-time data is quickly processed to provide real-time insights and decision support.

[0063] Step S20, performing integrated real-time computing processing on the source data through a real-time computing engine;

[0064] The data processing device of this embodiment can realize integrated processing of batch and stream data through a real-time computing engine, and reduce the delay from data generation to analysis through a batch and stream fusion data solution, thereby effectively solving the timeliness problem of batch data.

[0065] Among them, the real-time computing engine is an important tool in modern data processing, mainly used to process streaming data or real-time data. Real-time computing engines can be of various types, among which the main real-time computing engines can include:

[0066] Apache Flink: Flink is an open source real-time computing engine for distributed stream processing and batch processing. It is known for its low latency, high throughput, and exactly-once semantics, and is suitable for a variety of real-time data processing scenarios. Flink provides core concepts such as data streams, jobs, windows, and states, and supports time semantics (event time, processing time, ingestion time) to process data.

[0067] Spark Streaming: Spark Streaming is a component of Spark that processes real-time data streams. It is based on the principle of micro-batch processing, which divides the input data into certain time intervals and then processes them by the Spark engine. Although its processing delay is relatively large, usually in the order of seconds, it is still suitable for real-time data processing.

[0068] Storm: Storm is a distributed real-time computing system that can easily and reliably process unbounded data streams. It is also a very important real-time computing engine.

[0069] Serverless real-time computing engine: With the development of technology, real-time computing engines based on Serverless architecture have also begun to attract attention. For example, Alibaba Cloud provides a new generation of Serverless real-time computing engine, which is mainly aimed at real-time scenarios such as online applications, machine learning, microservices, and real-time risk control.

[0070] Each of the above engines has its own characteristics and is suitable for different data processing scenarios. For example, Flink and Spark Streaming have their own advantages in processing unbounded and bounded data streams, while the Serverless real-time computing engine provides more flexible resource management and expansion capabilities. When choosing a real-time computing engine, you can decide based on specific application requirements and scenario characteristics.

[0071] This embodiment is mainly aimed at big data scenarios to meet higher requirements for real-time, efficiency, consistency, and data quality.

[0072] As a result, the real-time computing engine processes and analyzes the continuously flowing data, without the need to store the data for subsequent processing. The real-time data is quickly processed to provide real-time insights and decision support.

[0073] Step S30: providing the processed real-time streaming data to the downstream system through the file system.

[0074] Among them, the data processed by the real-time computing engine for integrated real-time computing of the source data can be written back to Kafka in real time, and landed in the file system in real time, and directly provided to offline downstream and / or real-time downstream for further analysis and processing.

[0075] Therefore, after the real-time computing engine performs integrated real-time computing processing on the source data, step S200 may be further included to write the processed real-time stream data back to Kafka in real time for real-time data storage.

[0076] Afterwards, the real-time streaming data stored in Kafka can be provided to real-time downstream systems and / or offline downstream systems through the file system.

[0077] Therefore, step S30, providing the processed real-time streaming data to the downstream system through the file system may include:

[0078] Step S301, providing the processed real-time streaming data to the real-time downstream system through the file system;

[0079] The processed real-time stream data stored in Kafka can be provided to the real-time downstream system through the file system.

[0080] Step S302: providing the processed real-time streaming data to an offline downstream system through a file system.

[0081] The processed real-time stream data stored in Kafka can be provided to offline downstream systems through the file system.

[0082] This embodiment uses the above scheme, specifically by obtaining the real-time incoming source data; performing integrated real-time computing processing on the source data through a real-time computing engine; and providing the processed real-time streaming data to the downstream system through a file system. The technical solution of this application uses a real-time computing engine to perform integrated real-time computing processing on the source data, so that a set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve data accuracy and reliability. In addition, the real-time computing engine is used to perform integrated real-time computing processing on the source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. Therefore, the technical solution of this application can meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2, before the step of providing the processed real-time streaming data to the offline downstream system through the file system in step S302, the process further includes step S300:

[0084] Step S300, performing data quality verification on the processed real-time stream data through a preset data quality verification mechanism;

[0085] If the check passes, step S302 is executed to provide the processed real-time streaming data to the real-time downstream system through the file system.

[0086] As an implementation method, the verified offline data may be stored in an offline data warehouse, and the offline stored data may be provided to an offline downstream system.

[0087] This embodiment takes into account that due to factors such as instability of real-time computing clusters or storage clusters, real-time data may be lost or data association may fail during processing. Therefore, data quality verification is required before it can be provided to downstream data services.

[0088] Therefore, this embodiment introduces a data quality verification mechanism to perform data quality verification on the processed real-time streaming data.

[0089] Among them, the data quality inspection mechanism is the key link to ensure that the data is accurate, complete, consistent and timely. The level of data quality represents the degree to which the data meets the needs.

[0090] As an implementation method, the data quality inspection mechanism generally includes: data quality dimensions, data quality inspection architecture, data inspection rules, data quality analysis, etc. The above aspects are described in detail with examples below.

[0091] Data quality is usually measured from the following six dimensions:

[0092] Completeness: A measure of the completeness of the required data.

[0093] Uniqueness: Ensure that there are no duplicate values ​​for a data item or group of data.

[0094] Timeliness: Ensure that data is updated in a timely manner in response to users' requirements for timeliness in obtaining information.

[0095] Validity: The value and format of data must comply with the requirements of data definition or business definition.

[0096] Accuracy: Ensure that data reflects real business content.

[0097] Consistency: The type and meaning of data elements must be consistent and clear.

[0098] Among them, the data quality inspection architecture usually includes the following levels:

[0099] Integrity check: Check whether the data is complete and whether the field is empty or NULL.

[0100] Uniqueness test: Check whether the data is unique and whether there is duplicate data.

[0101] Consistency check: Check whether the data complies with consistency rules, such as value range constraints, format specifications, etc.

[0102] Timeliness test: Check whether the data is updated in a timely manner and whether there are problems such as missing time slices.

[0103] Accuracy test: Check whether the data is accurate and reliable, and whether there are any abnormal fluctuations.

[0104] Standardization verification: Check whether the data meets the standard requirements, such as coding rules, naming conventions, etc.

[0105] Among them, data verification rules usually include:

[0106] Integrity check rules: including empty or invalid value checks, record number abnormality checks, etc.

[0107] Uniqueness check rules: including duplicate data check, redundant data check, etc.

[0108] Consistency check rules: including mapping exception check, foreign key check, cross-validation rules, etc.

[0109] Timeliness check rules: including time slice missing check, etc.

[0110] Accuracy check rules: including value range anomaly check, time series fluctuation anomaly check, etc.

[0111] Standardization verification rules: including code value standardization checks, etc.

[0112] The main task of data quality analysis is to check whether there is dirty data in the data, including default values, abnormal values, inconsistent values, duplicate data, and data containing special symbols. These analyses help identify the causes, impacts, and solutions to data problems.

[0113] Through the above-mentioned data quality verification mechanism, the overall quality of data can be effectively guaranteed, thereby supporting the enterprise's data-driven decision-making and business processes.

[0114] In addition, there are a variety of tools to choose from for data quality inspection. Here are some commonly used open source data quality management tools:

[0115] Apache Griffin: Open sourced by eBay, it supports both batch and streaming data quality detection methods. It is a data quality service platform built on Hadoop and Spark, providing functions such as defining data quality models, performing data quality measurements, and automating data analysis and verification. The architecture of Apache Griffin is divided into three parts: Define, Measure, and Analyze, which are responsible for defining the dimensions of data quality statistics, performing statistical tasks, and saving and displaying statistical results.

[0116] Deequ: is a tool for data quality checking.

[0117] Great Expectations: This is an open source data quality tool for data validation and improving data quality.

[0118] Qualitis: This is a data quality management platform that supports quality verification, notification, and management services for multiple heterogeneous data sources. It is used to solve various data quality problems in business system operation, data center construction, and data governance.

[0119] DataCleaner: This is an open source tool for data quality analysis. It supports a variety of data sources, including file data, Hadoop, Hive, MongoDB, Neo4J, and traditional relational databases. After connecting to the data source, DataCleaner can quickly analyze the data table and generate reports for each field.

[0120] MobyDQ: This is a tool used by data engineering teams to automate data quality checks on data pipelines, catch data quality issues and trigger alerts in case of anomalies.

[0121] The above data quality inspection tools have their own characteristics and are suitable for different data quality inspection needs and environments. When choosing the right tool, you can consider factors such as specific project requirements, the team's technology stack, and budget.

[0122] The data quality inspection tool involved in the data quality inspection mechanism adopted in this embodiment is mainly used in a big data environment.

[0123] Among them, in the big data environment, it is very important to choose the right data quality inspection tool. The following data quality management tools suitable for the big data environment can be used:

[0124] Apache Griffin: Griffin is an open source big data quality solution developed by eBay, which supports both batch and streaming data quality detection methods. It is built on Hadoop and Spark, can process large-scale data sets, and provides data quality model definition, data quality measurement execution, automated data analysis and verification, etc.

[0125] Deequ: Deequ is a tool designed for large-scale data processing, compatible with Apache Spark, and suitable for data quality checking in big data environments. It helps users define data quality constraints and automatically validates these constraints on Spark datasets.

[0126] Great Expectations: Great Expectations is an open source data quality tool that can be used in big data environments and supports multiple data storage solutions, including big data platforms such as Hadoop and Spark. It is used for data validation and improving data quality, and can help teams establish and maintain data quality on large-scale data sets.

[0127] Qualitis: Qualitis is a data quality management platform that supports quality verification, notification, and management services for multiple heterogeneous data sources. It is suitable for big data environments and can handle complex data quality issues from different data sources.

[0128] All of the above tools have the ability to run efficiently in a big data environment, can process large-scale data sets, and support distributed computing frameworks such as Hadoop and Spark, thus meeting the needs of data quality inspection in a big data environment. When selecting a tool, factors such as the tool's scalability, compatibility, performance, and community support should be considered.

[0129] This embodiment uses the above scheme, specifically by obtaining the real-time inflow source data; performing integrated real-time calculation processing on the source data through the real-time calculation engine; and providing the processed real-time stream data to the downstream system through the file system. The technical solution of this application uses a real-time calculation engine to perform integrated real-time calculation processing on the source data, so that a set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, the real-time calculation engine is used to perform integrated real-time calculation processing on the source data, providing a solution for the fusion of batch data and real-time stream data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. Therefore, the technical solution of this application can meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios. In addition, before the technical solution of this application provides the real-time stream data stored in Kafka to the offline downstream system through the file system, the processed real-time stream data can also be verified by a preset data quality verification mechanism; if the verification is passed, the offline data of the verified data is stored in the offline data warehouse, and the offline stored data is provided to the offline downstream system, that is, it can be provided to the downstream for data service only after the data quality verification, which further guarantees the data quality.

[0130] Based on the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the second embodiment can be referred to the above introduction, and will not be repeated in the following. Figure 3 In step S300, after the processed real-time stream data is subjected to data quality verification through a preset data quality verification mechanism, if the verification fails, step S303 is also included:

[0131] Step S303, automatically triggering the preset backup link to process the source data, and executing step S302, providing the processed real-time streaming data to the offline downstream system through the file system.

[0132] Based on the above data quality inspection mechanism, this embodiment introduces a fallback link, namely an offline processing link, when the data quality inspection fails.

[0133] Wherein, if the inspection fails, the step of automatically triggering a preset backup link to process the source data includes:

[0134] If the test fails, the preset backup link is automatically triggered to perform batch calculations on the source data through a batch processing engine, and the batch calculated data is stored in an offline data warehouse and provided to an offline downstream system.

[0135] That is, when the real-time data verification fails, the execution of the backup link will be triggered, that is, the source data will be batch processed through a batch computing engine such as Hadoop and then stored in an offline data warehouse to prevent the timeliness of data services from being affected, while also further ensuring data quality.

[0136] This embodiment uses the above scheme, specifically by obtaining the real-time incoming source data; performing integrated real-time computing processing on the source data through a real-time computing engine; and providing the processed real-time streaming data to the downstream system through a file system. The technical solution of this application uses a real-time computing engine to perform integrated real-time computing processing on the source data, so that a set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve data accuracy and reliability. In addition, the real-time computing engine is used to perform integrated real-time computing processing on the source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. Therefore, the technical solution of this application can meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios. In addition, before the technical solution of the present application provides the real-time streaming data stored in Kafka to the offline downstream system through the file system, the processed real-time streaming data can also be verified for data quality through a preset data quality verification mechanism; if the verification passes, the offline data that passes the verification is stored in the offline data warehouse, and the offline stored data is provided to the offline downstream system; if the verification fails, the preset backup link is automatically triggered to process the source data, thereby introducing batch computing processing as a backup link to further ensure data quality.

[0137] Compared with traditional data processing architectures, batch data and real-time streaming data are often processed independently, but this often leads to data consistency problems and significantly increases the complexity and cost of development and operation and maintenance. In addition, this approach does not effectively solve the timeliness problem of batch data. In order to meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios, the embodiment of this application provides a batch-stream fusion data solution. The specific process can be referred to Figure 4 shown.

[0138] like Figure 4 As shown, this application includes the following contents:

[0139] Integrated processing of batch and stream data: Batch and stream data are uniformly processed using a real-time computing engine. The real-time computing engine is used to process and analyze continuously flowing data without storing the data for subsequent processing. Real-time data is quickly processed to provide real-time insights and decision support. The processed data can be written back to Kafka and other data sources in real time, and landed in the file system in real time, and directly provided to offline and real-time downstream for further analysis and processing.

[0140] Data quality verification mechanism: Due to factors such as the instability of real-time computing clusters or storage clusters, real-time data may be lost or unable to associate data during processing. Therefore, data quality verification is required before it can be provided to downstream data services.

[0141] Introducing a backup link: If the real-time data quality check fails, this technical solution will automatically trigger the backup link to process the data through batch calculation, further ensuring data quality.

[0142] The batch-stream fusion data solution mentioned above can reduce the delay from data generation to analysis, support rapid decision-making, and improve competitiveness. Using a set of development tools and frameworks to process data can reduce the complexity of development and operations, reduce the risk of data inconsistency, and improve data accuracy and reliability. In addition, this technical solution introduces batch computing processing as a backup link to further ensure data quality.

[0143] In addition, to achieve the above purpose, the present application also proposes a data processing device, such as Figure 5 As shown, the device comprises:

[0144] The acquisition module 10 is used to acquire the real-time inflow source data;

[0145] A calculation module 20, used for performing integrated real-time calculation processing on the source data through a real-time calculation engine;

[0146] The transmission module 30 is used to provide the processed real-time streaming data to the downstream system through the file system.

[0147] Among them, real-time computing and processing is mainly used to process and analyze continuously flowing data without the need to store the data for subsequent processing. Real-time data can be quickly processed to provide real-time insights and decision support.

[0148] In addition, the data processing device of this embodiment may further include a real-time data storage module, an offline data storage module and a backup link, wherein:

[0149] The real-time data storage module is used to store and manage the continuously flowing data that needs to be processed immediately or in near real time. It can quickly receive, store and query large amounts of streaming data for real-time analysis and decision-making.

[0150] Offline data storage modules are mainly used to process non-real-time data, such as historical data, batch data import, or big data analysis. These storage systems are usually designed to support the writing, storage, and batch processing of large-scale data.

[0151] Backup link. Since the instability of real-time computing clusters and storage clusters may cause problems such as missing real-time processing data, this solution introduces a data quality inspection mechanism, and on this basis, introduces a backup link, namely an offline processing link. When the real-time data verification fails, the execution of the backup link will be triggered, that is, the source data will be batch processed by a batch computing engine such as Hadoop and then stored in an offline data warehouse to prevent the timeliness of data services from being affected, and to further ensure data quality.

[0152] The data processing device provided by the present application adopts the data processing method in the above embodiment, and can use a set of development tools and frameworks to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, a real-time computing engine is used to perform integrated real-time computing and processing on source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided in the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0153] In addition, to achieve the above objectives, the present application also proposes a data processing system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0154] Reference below Figure 6, which shows a schematic diagram of the structure of a data processing system suitable for implementing the embodiment of the present application. The data processing system in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The data processing system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0155] like Figure 6 As shown, the data processing system may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing system are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. Communication device 1009 can allow the data processing system to communicate with other devices wirelessly or by wire to exchange data. Although the data processing system with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0156] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0157] The data processing system provided by the present application adopts the data processing method in the above-mentioned embodiment, and can use a set of development tools and frameworks to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, a real-time computing engine is used to perform integrated real-time computing and processing on source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. In addition, this technical solution introduces batch computing and processing as a bottom-up link to further ensure data quality. Compared with the prior art, the beneficial effects of the data processing system provided by the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and other technical features in the system are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0158] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0159] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0160] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0161] The computer-readable storage medium may be included in the data processing system; or may exist independently without being assembled into the data processing system.

[0162] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data processing system, the data processing system: obtains the real-time incoming source data; performs integrated real-time computing processing on the source data through the real-time computing engine; and provides the processed real-time streaming data to the downstream system through the file system.

[0163] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0166] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method. A set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, a real-time computing engine is used to perform integrated real-time computing and processing on source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness; in addition, this technical solution introduces batch computing and processing as a bottom-up link to further ensure data quality. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data processing method provided by the above-mentioned embodiment, which will not be repeated here.

[0167] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0168] The computer program product provided by this application can use a set of development tools and frameworks to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, improve the accuracy and reliability of data, and use a real-time computing engine to perform integrated real-time computing and processing on source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness; in addition, this technical solution introduces batch computing and processing as a bottom-up link to further ensure data quality. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, which will not be repeated here.

[0169] One or more technical solutions proposed in this application have at least the following technical effects: by obtaining the real-time inflow source data; performing integrated real-time computing processing on the source data through a real-time computing engine; and providing the processed real-time streaming data to the downstream system through a file system. The technical solution of this application uses a real-time computing engine to perform integrated real-time computing processing on the source data, so that a set of development tools and frameworks can be used to process data, reduce the complexity of development and operation, reduce the risk of data inconsistency, and improve the accuracy and reliability of data. In addition, the real-time computing engine is used to perform integrated real-time computing processing on the source data, providing a solution for the fusion of batch data and real-time streaming data, reducing the delay from data generation to analysis, supporting rapid decision-making, and improving competitiveness. Therefore, the technical solution of this application can meet the higher requirements for real-time, efficiency, consistency, and data quality in big data scenarios. In addition, this technical solution introduces batch computing processing as a bottom-up link to further ensure data quality.

[0170] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Get real-time incoming source data; Performing integrated real-time computing and processing on the source data through a real-time computing engine; The processed real-time streaming data is provided to the downstream system through the file system.

2. The method according to claim 1, characterized in that After the step of performing integrated real-time computing processing on the source data by the real-time computing engine, the method further includes: The processed real-time stream data is written back to Kafka in real time for real-time data storage.

3. The method according to claim 2, characterized in that The step of providing the processed real-time streaming data to the downstream system through the file system comprises: Provide real-time streaming data stored in Kafka to real-time downstream systems and / or offline downstream systems through the file system.

4. The method according to claim 3, characterized in that Before providing the real-time streaming data stored in Kafka to the offline downstream system through the file system, it also includes: The processed real-time streaming data is verified for data quality through a preset data quality verification mechanism; If the verification is passed, the offline data that has passed the verification will be stored in the offline data warehouse, and the offline stored data will be provided to the offline downstream system.

5. The method according to claim 4, characterized in that After the step of performing data quality verification on the processed real-time stream data through a preset data quality verification mechanism, the method further includes: If the test fails, the preset backup link is automatically triggered to process the source data.

6. The method according to claim 5, characterized in that If the inspection fails, the step of automatically triggering a preset backup link to process the source data includes: If the test fails, the preset backup link is automatically triggered to perform batch calculations on the source data through a batch processing engine, and the batch calculated data is stored in an offline data warehouse and provided to an offline downstream system.

7. A data processing device, characterized in that: The device comprises: The acquisition module is used to obtain the real-time incoming source data; A computing module, used for performing integrated real-time computing processing on the source data through a real-time computing engine; The transmission module is used to provide the processed real-time streaming data to the downstream system through the file system.

8. A data processing system, characterized in that: The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images