A large data stream and batch integrated processing system and its working method

Abstract

The present application discloses a system based on integrated processing of big data streams and batches and a working method thereof. The working method of the system is as follows: obtaining data to be processed, where the data comes from a network, a business application unit or historical data stored in a data warehouse and a computing unit; transmitting the data to be processed to the data warehouse and the computing unit through a distributed queue message, and at the same time, the background management system transmits processing parameters; the data warehouse and the computing unit obtain the data to be processed, and analyze it according to the processing parameters to obtain analysis results; the analysis results are stored in a distributed storage unit; and the business application unit displays the analysis results in the distributed storage service to the client according to the client's request.

Description

Technical Field

[0001] The present application relates to the field of big data, and specifically to a big data stream-batch integrated processing system and its working method. Background Art

[0002] With the rapid development of information technology, nearly every industry is embracing digitalization, ushering in the era of the digital economy. The digital age also means the era of data. As the digital age deepens, the types of data generated are multiplying, and the volume of data is growing exponentially. To leverage this data to better promote social development, it requires processing, storage, and analysis. However, the sheer scale of data presents certain challenges to both storage and processing.

[0003] When the big data industry first emerged, data processing and storage mostly relied on HDFS (Hadoop Distributed File System) and MapReduce (a programming framework for distributed computing programs), and data processing was mainly in the form of batch processing. With the development of the industry and combined with actual applications, the form of data processing is no longer limited to batch processing, but a new data processing idea, stream processing, has been introduced. Currently popular stream processing frameworks (such as Spark-streaming and Flink) are relatively mature stream processing frameworks. Data storage has also evolved from a single relational database to Nosql databases, key-value storage databases, object storage databases, and time series databases, etc. Today, data sources are extensive and diverse, and only by effectively combining batch processing and stream processing methods can we meet the growing demand.

[0004] As society develops in the future, more data will be generated. However, simply looking at the surface of this data will prevent us from discovering the important information it holds. Only by processing the data through analytical modeling can we achieve the desired results. Therefore, a model designed and implemented for big data systems is needed to process and analyze this data, further leveraging the potential of these data assets and empowering social development. Based on this model, a reliable big data storage, processing, and visualization system can be quickly established. This system model can integrate and process data from related fields, dynamically expand data processing algorithms, and simultaneously meet the needs of real-time and historical data processing, maximizing system functionality and data effectiveness.

[0005] Therefore, this application proposes a big data stream and batch integrated processing system and its working method, which can be used to design and implement big data storage and processing systems in related fields, making it easier for data owners to realize their own data decision-making centers. Summary of the Invention

[0006] The purpose of this application is to provide a big data stream and batch integrated processing system and its working method, which solves the problem in the existing technology that stream processing and batch processing of big data cannot be effectively combined, and designs, optimizes and expands the data collection, processing, storage and display processes, making the development of big data application systems more standardized and efficient.

[0007] This application is implemented through the following technical solutions:

[0008] In a first aspect, the present application provides a system for integrated big data stream and batch processing, comprising an online analytical processing (OLAP) module for back-end data processing and an online transaction processing (OLTP) module for front-end services, wherein the OLAP module and the OLTP module are interconnected; the OLAP module comprises a data forwarding service unit, a back-end management unit, and a data warehouse and computing unit; and the OLTP module comprises a distributed storage unit, a business application unit, and a client.

[0009] The data forwarding service unit, the background management unit, the data warehouse and the computing unit, the distributed storage unit and the business application unit are connected to each other through the distributed message queue unit, and the data warehouse and the computing unit are respectively connected to the background management unit and the distributed storage unit;

[0010] The data forwarding service unit is used to obtain the data to be processed transmitted by the external system and the business application unit, and transmit the data to be processed to the distributed message queue unit, wherein the data to be processed is stream data, and the stream data is used to represent real-time data;

[0011] The backend management unit is used to configure historical data, data processing algorithms, and data processing parameters, and directly transmit the historical data and data processing algorithms to the data warehouse and computing unit to achieve dynamic expansion of data and data processing algorithms; and send the data processing parameters to the data warehouse and computing unit through the distributed message queue unit to achieve dynamic configuration of data processing parameters; obtain the analysis results of the data by the data warehouse and computing unit, and visualize the analysis results;

[0012] The data warehouse and computing unit are used to obtain the to-be-processed data and data processing parameters in the distributed message queue unit, and perform stream processing analysis on the to-be-processed data in combination with the data processing algorithm issued by the background management unit to obtain stream data analysis results; and obtain historical data stored in the data warehouse and computing unit, and perform batch processing analysis on the historical data according to the data processing parameters and the data processing algorithm to obtain batch data analysis results; the stream data analysis results and the batch data analysis results are jointly used as analysis results, and the analysis results are transmitted to the background management unit and the distributed storage unit;

[0013] The distributed storage unit is used to store the analysis results and transmit the analysis results to the business application unit through the distributed storage unit;

[0014] The business application unit is used to carry system business and transmit the data analysis results in the distributed storage unit to the client.

[0015] In a possible implementation, obtaining the data to be processed transmitted by the external system and the business application unit includes: obtaining the data to be processed transmitted by the external system and the business application unit through network communication, serial communication, and satellite communication.

[0016] In a possible embodiment, a Mysql database is also provided on the background management unit. The background management unit transmits the data processing parameters generated in response to human-computer interaction to the distributed message queue unit or stores them in the Mysql database. When the data warehouse and the computing unit process the data to be processed, the data processing parameters in the Mysql database are scheduled for processing.

[0017] In one possible implementation, the data warehouse and computing unit include a Hadoop cluster, an Hbase cluster, a Hive cluster, and a Flink cluster;

[0018] The Hadoop cluster uses its distributed file system HDFS to store unstructured data; the Hbase cluster stores structured subject data; the Flink cluster analyzes and processes the unstructured data in the distributed file system HDFS, the structured subject data in the Hbase cluster, and the data to be processed transmitted by the distributed message queue unit to obtain analysis results, and stores the analysis results in the Mysql database of the background management unit or the distributed storage unit respectively; the Hive cluster analyzes and processes the structured subject data in the Hbase cluster to obtain analysis results, and stores the analysis results in the Mysql database of the background management unit or the distributed storage unit respectively.

[0019] In a possible implementation, the distributed message queuing unit uses a Kafka cluster to implement a distributed message queue, meet high data throughput, and complete data buffering.

[0020] In a possible implementation, the distributed storage unit includes a distributed database, a FastDFS cluster, and a Redis cluster; the distributed database includes a MyCat cluster and a Mysql cluster to implement distributed database and high concurrency and provide expansion support.

[0021] In a possible implementation, the business application unit includes a microservice subunit and a load balancing unit. The load balancing unit is used to implement load-balanced access for multiple users, and the microservice subunit is used to carry system services and transmit access-acquired data to the client.

[0022] In a second aspect, the present application provides a working method based on a large data stream and batch integrated processing system, including a stream data processing process and a batch data processing process;

[0023] The stream data processing process includes:

[0024] The backend management unit sends the data processing algorithm to the data warehouse and the computing unit, obtains the stream data to be processed through the data forwarding service unit, and transmits the stream data to be processed to the distributed message queue unit for queuing. At the same time, the backend management unit sends the data processing parameters corresponding to the stream data to be processed to the distributed message queue unit for queuing;

[0025] The data warehouse and computing unit obtain the stream data to be processed, the data processing algorithm, and the data processing parameters. Combined with the data processing algorithm and data processing parameters, the Flink cluster in the data warehouse and computing unit analyzes and processes the stream data to be processed, obtaining the structured topic data corresponding to the stream data to be processed and the stream data analysis results.

[0026] The structured subject data is stored in the Hbase cluster, and the stream data analysis results are transmitted to the MySQL database of the background management unit and the distributed storage unit respectively;

[0027] The stream data analysis results are transmitted to the business application unit through the distributed storage unit, and the business application unit forwards the stream data analysis results to the client for visualization, completing the stream data processing;

[0028] The batch data processing process includes:

[0029] Send historical data and data processing algorithms to the data warehouse and computing unit through the background management unit, and send data processing parameters to the MySQL database through the background management unit;

[0030] Read historical data from the HBase cluster and the distributed file system HDFS through the Flink cluster, or obtain historical data issued by the background management unit through the Flink cluster. Combined with the data processing algorithm and the data processing parameters in the MySQL database of the background management unit, the historical data is processed and analyzed to obtain batch data analysis results, which are then stored in the distributed storage unit or MySQL database.

[0031] The batch data analysis results are transmitted to the business application unit through the distributed storage unit, and the business application unit forwards the batch data analysis results to the client for visualization, or the batch data analysis results in the Mysql database are visualized through the background management unit to complete the batch data processing.

[0032] In one possible implementation, the Flink cluster in the data warehouse and computing unit performs analysis and processing on the stream data to be processed, including:

[0033] When the Flink cluster starts working for the first time, the default data processing parameters in the MySQL database of the background management unit are read through the Flink cluster.

[0034] Analyze and process the stream data to be processed according to the default data processing parameters;

[0035] When the data processing parameters need to be changed, the dynamic data processing parameters are sent to the distributed message queue unit through the background management unit, and the dynamic data processing parameters are forwarded to the data warehouse and computing unit through the distributed message queue unit to realize dynamic parameter configuration.

[0036] In a possible implementation, the method further includes converting batch data into stream data, and performing fusion analysis on the original stream data and the converted stream data through a Flink cluster.

[0037] This application provides a big data stream and batch integrated processing system, which realizes stream processing, batch processing, and fusion processing of big data through a big data storage and processing framework, and realizes the dynamic expansion of processing data and processing algorithms. It also optimizes the entire system design by combining system design concepts such as OLAP, OLTP, and microservices, thereby achieving the effect of easy implementation, complete functionality, and high scalability of the big data system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present application, the following briefly introduces the drawings required for use in the examples. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be considered as limiting the scope. A person of ordinary skill in the art can also derive other relevant drawings based on these drawings without inventive effort. In the drawings:

[0039] Figure 1 A schematic diagram of the structure of a large data stream and batch integrated processing system provided in an embodiment of the present application.

[0040] Figure 2 A schematic diagram of the flow processing process provided in an embodiment of the present application.

[0041] Figure 3 This is a schematic diagram of Flink's operation during the stream processing process provided in an embodiment of the present application.

[0042] Figure 4 Schematic diagram of the batch processing process provided in the embodiment of the present application.

[0043] Figure 5 A schematic diagram of batch processing and stream processing conversion provided in an embodiment of the present application.

[0044] Figure 6 A schematic diagram of a data management process provided in an embodiment of the present application.

[0045] Figure 7 A schematic diagram of a data processing management process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of this application more clear, the present application is further described in detail below in conjunction with examples and drawings. The schematic implementation methods of this application and their descriptions are only used to explain this application and are not intended to limit this application.

[0047] Example 1

[0048] like Figure 1 As shown, the present application provides a large data stream and batch integrated processing system, including an online analytical processing module OLAP for back-end data processing and an online transaction processing module OLTP for front-end services. The online analytical processing module OLAP and the online transaction processing module OLTP are interconnected; the online analytical processing module OLAP includes a data forwarding service unit, a background management unit, and a data warehouse and computing unit, and the online transaction processing module OLTP includes a distributed storage unit, a business application unit, and a client.

[0049] The data forwarding service unit, the background management unit, the data warehouse and computing unit, the distributed storage unit and the business application unit are interconnected through the distributed message queue unit, and the data warehouse and computing unit are respectively connected to the background management unit and the distributed storage unit.

[0050] The data forwarding service unit is used to obtain the data to be processed transmitted by the external system and business application unit, and transmit the data to be processed to the distributed message queue unit. The data to be processed is stream data, which is used to represent real-time data.

[0051] In actual production, data access methods are complex, such as accessing data through TCP Client / Server, WebService, web crawlers, email, SMS, etc. There are many different communication methods, such as common network communication, serial port communication, satellite communication, etc. The sources of data are different, such as external system data, equipment collection data, system self-generated data, etc. Due to the complexity and diversity of collected data, it is necessary to design dedicated data collection services to facilitate data access management and expansion.

[0052] The background management unit is used to configure historical data, data processing algorithms and data processing parameters, and transmit historical data and data processing algorithms directly to the data warehouse and computing unit to achieve dynamic expansion of data and data processing algorithms; and send data processing parameters to the data warehouse and computing unit through the distributed message queue unit to achieve dynamic configuration of data processing parameters; obtain the analysis results of the data by the data warehouse and computing unit, and visualize the analysis results.

[0053] The background management unit is mainly aimed at system development and operation and maintenance personnel to complete system management and operation and maintenance. Through the background management unit module, operation and maintenance personnel can monitor the operating status of the system software and hardware, perform some data operations, configure parameters for data processing, and dynamically add data processing algorithms.

[0054] The data warehouse and computing unit are used to obtain the data to be processed and the data processing parameters in the distributed message queue unit, and perform stream processing analysis on the data to be processed in combination with the data processing algorithm issued by the background management unit to obtain the stream data analysis results; and obtain the historical data stored in the data warehouse and the computing unit, and perform batch processing analysis on the historical data according to the data processing parameters and the data processing algorithm to obtain the batch data analysis results; the stream data analysis results and the batch data analysis results are jointly used as the analysis results, and the analysis results are transmitted to the background management unit and the distributed storage unit.

[0055] The data warehouse and computing unit are the core modules of big data storage and processing, including distributed computing storage frameworks and resource scheduling frameworks such as Hadoop, HBase, Hive and Flink. They involve data access, data synchronization, parallel computing, stream processing and batch processing. The integrated data stream and batch processing model is realized through the coordination of component frameworks. At the same time, data storage and computing can be dynamically expanded.

[0056] The distributed storage unit is used to store the analysis results and transmit the analysis results to the business application unit through the distributed storage unit.

[0057] When targeting internet users, a standalone data service system alone cannot support the large number of concurrent users, requiring a distributed architecture to meet these demands. In the initial stages of system construction, a standalone business data storage service can be used. However, as the number of users increases over time and concurrent access becomes insufficient, the storage service will need to be expanded to a distributed cluster. Scalability should be a key consideration during system design.

[0058] The business application unit is used to carry system business and transmit the data analysis results in the distributed storage unit to the client.

[0059] The data warehouse and computing unit can obtain the data processing parameters stored in the Mysql database in the background management unit. However, if the data processing parameters need to be changed during the data processing process, they can only be transmitted to the data warehouse and computing unit through the message queue, thereby realizing the dynamic configuration of the data processing parameters and dynamically changing the parameters of the processing process without stopping the data processing.

[0060] The business application unit mainly implements a distributed application system for user access based on the relevant business involved in the system. It includes system basic microservices such as API gateway, service registration center, user authentication and configuration center, as well as various microservices related to the system business. At the same time, it connects to the distributed storage module to support the expansion of concurrent access.

[0061] In more extreme cases, when the system streaming data access reaches hundreds of thousands per second, ordinary data receiving services may not be able to receive such a large amount of data within a unit of time, resulting in data loss. To meet the high throughput requirements of system data access, Kafka message queues are used, combined with data forwarding service load balancing to achieve timely response to large-scale data access. At the same time, Kafka can also be used as a data cache service, retaining real-time data even when the background system crashes, to ensure that data is not lost.

[0062] In a possible implementation, obtaining the data to be processed transmitted by the external system and the business application unit includes: obtaining the data to be processed transmitted by the external system and the business application unit through network communication, serial communication, and satellite communication.

[0063] In one possible implementation, a Mysql database is also provided on the background management unit. The background management unit transmits the data processing parameters generated in response to human-computer interaction to the distributed message queue unit or stores them in the Mysql database. When the data warehouse and the computing unit process the data to be processed, the data processing parameters in the Mysql database are scheduled for processing.

[0064] The backend management unit serves both business systems and data analysis. For business systems, it configures data display and manages business data resources. For data analysis, it imports and exports data, configures data processing parameters, manages data processing algorithms, views and analyzes data reports, and monitors the operational status of the entire online analytical processing (OLAP) module.

[0065] In one possible implementation, the data warehouse and computing unit include a Hadoop cluster, an Hbase cluster, a Hive cluster, and a Flink cluster;

[0066] The Hadoop cluster uses its distributed file system HDFS to store unstructured data, Mapreduce to implement offline data analysis, and Yarn to implement resource scheduling and task management; the Hbase cluster stores structured topic data; the Flink cluster analyzes and processes the unstructured data in the distributed file system HDFS, the structured topic data in the Hbase cluster, and the unprocessed data transmitted by the distributed message queue unit, obtains analysis results, and stores the analysis results in the MySQL database of the background management unit or the distributed storage unit; the Hive cluster analyzes and processes the structured topic data in the Hbase cluster, obtains analysis results, and stores the analysis results in the MySQL database of the background management unit or the distributed storage unit.

[0067] Hadoop clusters use Hadoop HDFS to store data, MapReduce for offline data analysis, and Yarn for resource scheduling and task management. Hbase clusters implement massive data storage and distributed queries. Hive clusters implement offline data analysis, using SQL-like statements (hql) to complete data analysis. Flink clusters implement real-time data streaming processing and historical data batch processing. Kafka clusters implement distributed message queues to meet high data throughput and complete data buffering. MyCat+MySQL clusters implement distributed databases and provide expansion support for high concurrency. FastDFS clusters implement distributed small file systems to store small files, JSON data, image data, etc., and provide expansion support for high concurrency. Redis clusters implement distributed caching to improve system operation efficiency and provide expansion support for high concurrency.

[0068] Data processing must be coordinated with data access and data storage, and appropriate processing procedures must be formulated based on specific business functions. The two approaches to big data processing are batch processing and stream processing.

[0069] Based on time analysis, data can be divided into real-time data and historical data. Real-time data is generally collected by device terminals at the current point in time, such as data collected by sensors, data generated by system client operations, or data collected by related service software. Historical data can be understood as archived storage of real-time data; once the data is stored in the system, it becomes historical data. If reports or related data sets require real-time display, stream processing should be used to process real-time data. If real-time display is not required, the decision to use batch processing or real-time stream processing of historical data should be made based on the frequency of access, the complexity of data processing, and system performance considerations. Stream processing is a continuous, non-stop processing process that consumes system resources at all times. Batch processing is a one-time process that consumes system resources only during processing, and the resource usage is much higher than that of stream processing. Furthermore, larger data volumes require longer processing times. According to spatial analysis, data is divided into streaming data and datasets. Streaming data is data transmitted in communication networks, and datasets are persistent data. Streaming data processing is suitable for stream processing, while datasets are suitable for batch processing. Streaming data and datasets can be converted into each other. Therefore, during data processing, the data can be converted according to the situation and the corresponding processing method can be selected.

[0070] Data processing results are stored or cached based on the data service system. Backend systems do not require concurrency, only storage access is required. The frontend has higher requirements for concurrent access and uses cache. If data processing results need to be persistent, they need to be stored in a database and an update cache needs to be established.

[0071] After data processing is completed, it needs to be visualized. According to the final output of the data processing, charts are selected for data visualization. There are many open source data visualization libraries with detailed usage documentation and examples, such as: Cesium, Echarts and D3, which are easy to use and powerful.

[0072] In a possible implementation, the distributed message queuing unit uses a Kafka cluster to implement a distributed message queue, meet high data throughput, and complete data buffering.

[0073] In a possible implementation, the distributed storage unit includes a distributed database, a FastDFS cluster, and a Redis cluster; the distributed database includes a MyCat cluster and a Mysql cluster to implement distributed database and provide expansion support for high concurrency.

[0074] In one possible implementation, the business application unit includes a microservice subunit and a load balancing unit. The business system is designed and implemented according to the microservices philosophy, with a reasonable service split. Load balancing access should be implemented, and circuit breaking and degradation measures should be implemented. The load balancing unit is used to achieve load balancing access for multiple users, while the microservice subunit is used to carry system services and transmit access data to the client.

[0075] The business application unit is based on the microservice architecture. Starting from the system design and implementation, without considering the system business, the basic application services are as follows: API Gateway: provides a unified interface for access to each microservice; Registration Center: Service registration center, realizing registration and discovery of microservices; User Center: provides user-related services, user login and authentication, etc.; Configuration Center: provides a unified configuration center when the system involves a large number of microservices to facilitate service configuration; Service Monitoring: monitors the service operation status, especially when there are many services, this service must be implemented.

[0076] Example 2

[0077] This application provides a working method based on a large data stream and batch integrated processing system, including a stream data processing process and a batch data processing process;

[0078] like Figure 2 As shown in the figure, the stream data processing process includes:

[0079] The data processing algorithm is sent to the data warehouse and computing unit through the background management unit, the stream data to be processed is obtained through the data forwarding service unit, and the stream data to be processed is transmitted to the distributed message queue unit for queuing. At the same time, the background management unit sends the data processing parameters corresponding to the stream data to be processed to the distributed message queue unit for queuing.

[0080] For a complete data processing chain, the first considerations are the data source, how the data is generated, and how the generated data is transmitted to the backend server. Data comes from two common sources: data collected by devices and data from the network. Data can be transmitted via wireless communication networks, such as satellite communications, or wired networks. From a system performance perspective, data access services can serve as the load balancing component of system data access to meet the system's high data throughput requirements.

[0081] After determining the data source and data communication method, data access services can be implemented according to relevant data protocols. First, data access categories must be categorized based on the specific needs of the data access and the relevant business. Then, data access methods must be categorized. This categorization allows data to be accessed through different topics and partitions in the Kafka message queue, achieving load balancing and improving data access throughput. Data access service expansion, suspension, and modification should be recorded in the backend management unit to facilitate data access service management.

[0082] The data warehouse and computing unit are used to obtain the stream data to be processed, the data processing algorithm and the data processing parameters. The data processing algorithm and the data processing parameters are combined, and the Flink cluster in the data warehouse and computing unit is used to analyze and process the stream data to be processed, and the structured topic data corresponding to the stream data to be processed and the stream data analysis results are obtained.

[0083] The data analysis results are the final results of large-scale data analysis and can be directly used for data visualization. One type of analysis result is business-oriented data, which will be stored in the distributed database of the front-end service. Another type of analysis result is data for the back-end system, which will be stored in the Mysql database of the back-end service. The former serves business, and the latter serves system management decision-making.

[0084] Through data cleaning, conversion and other steps, the data is normalized to obtain structured topic data. Structured topic data is a large amount of data and can be used as the basic data for data analysis. This data is stored in Hbase for subsequent data analysis and query.

[0085] This structured topic data is stored in an HBase cluster, and the stream data analysis results are transmitted to the MySQL database in the backend management unit and the MySQL database in the distributed storage unit. After the data analysis results are stored in MySQL or the distributed storage unit, the relevant data needs to be visualized. For front-end systems, high concurrent user access can easily lead to system access bottlenecks. Adding cache can improve system performance, reduce access time, and provide a better user experience.

[0086] The stream data analysis results are transmitted to the business application unit through the distributed storage unit, and the business application unit forwards the stream data analysis results to the client for visualization, completing the processing of the stream data.

[0087] Optionally, the Flink cluster in the data warehouse and computing unit is used to analyze and process the stream data to be processed, including: when the Flink cluster works for the first time, the default data processing parameters in the Mysql database of the background management unit are read by the Flink cluster; according to the default data processing parameters, the stream data to be processed is analyzed and processed; when the data processing parameters need to be changed, the dynamic data processing parameters are sent to the distributed message queue unit through the background management unit, and the dynamic data processing parameters are forwarded to the data warehouse and computing unit through the distributed message queue unit to realize dynamic parameter configuration.

[0088] like Figure 3 As shown in the figure, for Flink stream processing programs, Kafka not only provides the data to be processed, but also provides information such as data processing parameters. The data processing parameters are sent to the Kafka message queue through the backend management system. Flink receives the data processing parameters through different topics and integrates them into the data processing. This process allows the configuration of some processing parameters without stopping the Flink service.

[0089] When the Flink stream processing task is started for the first time, it needs to load relevant data processing parameters from MySQL and store the parameters in the Map cache as default parameters for the Flink stream processing operator to access for data processing. When the parameters need to be adjusted, the dynamic parameters are sent to topic2 through the system background management service, and the MySQL parameter information is updated at the same time. The Flink stream processing operator obtains the dynamic parameter information and stores it in the Flink stream processing task state Mapstate. When the relevant parameters can be found in Mapstate, the Flink stream processing operator no longer accesses the Map cache, but accesses the parameter information in Mapstate. Finally, the Flink stream processing operator chain that completes data processing according to the streaming data processing algorithm completes the data processing and outputs the results.

[0090] like Figure 4 As shown in Figure 2, the batch data processing process includes:

[0091] The historical data in the Hbase cluster is read through the Hive cluster, and the historical data is processed and analyzed to obtain data analysis results; the data analysis results are stored in the Mysql database of the background management unit, and the data analysis results are visualized through the background management system.

[0092] The Flink cluster reads historical data from the Hbase cluster and the distributed file system HDFS, or obtains historical data issued by the background management unit through the Flink cluster. Combined with the data processing algorithm and the data processing parameters in the MySQL database of the background management unit, the historical data is processed and analyzed to obtain batch data analysis results, and the batch data analysis results are stored in the distributed storage unit or MySQL database; the batch data analysis results are transmitted to the business application unit through the distributed storage unit, and the business application unit forwards the batch data analysis results to the client for visualization, or the batch data analysis results in the MySQL database are visualized through the background management unit. Among them, the dotted part is the process of Hql analysis, which is described as follows:

[0093] (1) Establishing a mapping between Hbase and Hive

[0094] Hbase mainly performs data storage and distributed query and does not have data analysis capabilities. However, Hive has Hql statements similar to SQL statements for data query and statistical analysis. Hive also provides the ability to integrate Hbase through configuration. After that, you can analyze the data in Hbase by writing Hql statements.

[0095] (2) Complete the hql analysis statement and execute the data analysis statement through Hive

[0096] Hql statements are SQL-like statements with a low learning cost. Hive converts the Hql language into Mapreduce data batch processing programs at the bottom layer. Users do not need to pay attention to the underlying implementation and can use Hql statements to process and analyze data. It is very friendly for people who are not good at writing batch processing programs but need to perform data analysis.

[0097] Write hql statements according to data analysis requirements, and then make hql into shell scripts. When performing data analysis, you can directly execute the shell script to implement this type of data analysis. At the same time, you can also realize data processing automation, such as timed processing and analysis of data.

[0098] (3) Export data to MySQL

[0099] After data analysis is completed, the data is stored in Hive, and the data analysis results in Hive are synchronized to MySQL through the data synchronization service.

[0100] The solid line represents the process of batch analysis by Flink, which is described as follows:

[0101] (1) Determine the data source

[0102] The data processed by batch processing is generally local historical data. The local data stored in the system are HDFS and Hbase, so the data for data batch processing is obtained from HDFS and Hbase. If there is external historical data such as data files, you can first import HDFS as the data source through the background management system.

[0103] (2) Flink batch tasks obtain data for processing

[0104] Flink batch processing programs primarily perform data analysis and processing, reading HDFS data or HBase data for data analysis and processing of a single data source, or simultaneously reading data from both for fused data analysis and processing. Flink batch processing programs can also perform data cleaning and conversion, processing and storing unstructured data in HDFS in HBase.

[0105] like Figure 5 As shown, the working method also includes: converting batch data into stream data, and performing fusion analysis and processing on the original stream data and the converted stream data through the Flink cluster.

[0106] When designing the processing flow, you need to specify the data structure that will be input to the Flink stream processing task and reserve a Kafka data source for the Flink stream processing task. The Flink stream processing task receives data from channels topic 1 and topic 2, merges them, and then processes them.

[0107] The basic idea of ​​dynamic expansion of data processing is to decouple the data itself from data processing, and to simultaneously manage both data and data processing.

[0108] like Figure 6 As shown, this embodiment provides a data management process, which specifically includes:

[0109] (1) Management of data files.

[0110] Typically, external data is imported into the distributed file system HDFS for data analysis. After the data file is imported through the background management service, metadata information such as the file path, import time, and file description needs to be recorded in the MySQL database for data management and subsequent data processing.

[0111] (2) Data warehouse management.

[0112] Hbase stores streaming data from the system, data sets synchronized through HDFS, and some intermediate processing results. Too many data tables are very detrimental to data management. Therefore, when creating a table, you should enter the table structure, table function, table data description information, etc. through the background management service. When data analysis is required, you can view the table data information through the background service, which will greatly help data management and processing.

[0113] like Figure 7 As shown, this embodiment provides a data processing management process, including:

[0114] (1) Write a shell script to submit a Flink task

[0115] Write the shell command for submitting Flink tasks into a script, specifying the Flink package location and parameters such as the resource usage of the Flink task submission. Store the script in a fixed location on the Linux system so that it can be called by the backend management service. This step can be completed at the beginning of system construction. Subsequent data processing algorithm expansion can be directly called from the backend management system.

[0116] (2) Determine the data source

[0117] If it is a stream processing algorithm, you only need to focus on the data access service. Kafka is used as the data interface in the middle to realize data communication between the data access service and the Flink stream processing program.

[0118] If it is a batch processing algorithm, the data to be processed can be determined based on the data management part.

[0119] (3) Writing Flink stream processing / batch processing programs

[0120] Design and implement the Flink application based on the selected data source information, related data structure information, and data analysis requirements, and finally package the implemented Flink program.

[0121] (4) Upload Flink stream processing / batch processing program algorithm package

[0122] Use the backend management service to upload the packaged Flink application to the file system of the Linux server where the Flink cluster resides. The database records information such as the Flink application algorithm package path, description, and processing parameters.

[0123] (5) Create / Start / Stop Flink Tasks

[0124] Create a Flink task through the backend management service and store task information in MySQL, including the Flink application algorithm used. Use SSH remotely to call a shell script to start the Flink task. The returned YARN application ID is updated to the current Flink task. Use the YARN application ID and SSH remote calls to stop the corresponding Flink task, such as executing a command such as "stop the YARN task."

[0125] (6) Monitor the running status of the task

[0126] The backend management system obtains task status information through the API provided by YARN, updates the latest status information to the corresponding MySQL task, and establishes scheduled tasks to track task status in real time.

[0127] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of this application in detail. It should be understood that the above are only specific implementation methods of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A large data stream and batch integrated processing system, characterized in that: The system comprises an online analytical processing module (OLAP) for back-end data processing and an online transaction processing module (OLTP) for front-end services, wherein the OLAP and OLTP modules are interconnected. The OLAP module comprises a data forwarding service unit, a back-end management unit, and a data warehouse and computing unit, while the OLTP module comprises a distributed storage unit, a business application unit, and a client. The data forwarding service unit, the background management unit, the data warehouse and the computing unit, the distributed storage unit and the business application unit are connected to each other through the distributed message queue unit, and the data warehouse and the computing unit are respectively connected to the background management unit and the distributed storage unit; The data forwarding service unit is used to obtain the data to be processed transmitted by the external system and the business application unit, and transmit the data to be processed to the distributed message queue unit, wherein the data to be processed is stream data, and the stream data is used to represent real-time data; The background management unit is used to configure historical data, data processing algorithms and data processing parameters, and directly transmit historical data and data processing algorithms to the data warehouse and computing unit to achieve dynamic expansion of data and data processing algorithms; The data processing parameters are sent to the data warehouse and computing unit through the distributed message queue unit to achieve dynamic configuration of data processing parameters; the analysis results of the data warehouse and computing unit on the data are obtained and visualized; The data warehouse and computing unit are used to obtain the data to be processed and the data processing parameters in the distributed message queue unit, and perform stream processing analysis on the data to be processed in combination with the data processing algorithm issued by the background management unit to obtain the stream data analysis results; and obtaining historical data stored in the data warehouse and the computing unit, performing batch analysis on the historical data according to data processing parameters and data processing algorithms, and obtaining batch data analysis results; The stream data analysis result and the batch data analysis result are used together as the analysis result, and the analysis result is transmitted to the background management unit and the distributed storage unit; The distributed storage unit is used to store the analysis results and transmit the analysis results to the business application unit through the distributed storage unit; The business application unit is used to carry system business and transmit the data analysis results in the distributed storage unit to the client; The data warehouse and computing unit include Hadoop cluster, Hbase cluster, Hive cluster and Flink cluster; The Hadoop cluster uses its distributed file system HDFS to store unstructured data; the Hbase cluster stores structured subject data; the Flink cluster analyzes and processes the unstructured data in the distributed file system HDFS, the structured subject data in the Hbase cluster, and the data to be processed transmitted by the distributed message queue unit to obtain analysis results, and stores the analysis results in the Mysql database of the background management unit or the distributed storage unit respectively; the Hive cluster analyzes and processes the structured subject data in the Hbase cluster to obtain analysis results, and stores the analysis results in the Mysql database of the background management unit or the distributed storage unit respectively.

2. The large data stream and batch integrated processing system according to claim 1 is characterized in that: Acquiring data to be processed transmitted by external systems and business application units, including: acquiring data to be processed transmitted by external systems and business application units through network communication, serial port communication, and satellite communication.

3. The large data stream and batch integrated processing system according to claim 1 is characterized in that: The background management unit is also provided with a Mysql database. The background management unit transmits the data processing parameters generated in response to human-computer interaction to the distributed message queue unit or stores them in the Mysql database. When the data warehouse and the computing unit process the data to be processed, the data processing parameters in the Mysql database are scheduled for processing.

4. The large data stream and batch integrated processing system according to claim 3 is characterized in that: The distributed message queue unit adopts a Kafka cluster to implement a distributed message queue, meet high data throughput and complete data buffering.

5. The large data stream and batch integrated processing system according to claim 3 is characterized in that: The distributed storage unit includes a distributed database, a FastDFS cluster and a Redis cluster; the distributed database includes a MyCat cluster and a Mysql cluster to provide expansion support for the distributed database and high concurrency.

6. The large data stream and batch integrated processing system according to claim 3 is characterized in that: The business application unit includes a microservice subunit and a load balancing unit. The load balancing unit is used to achieve load balanced access for multiple users. The microservice subunit is used to carry system services and transmit access-acquired data to the client.

7. A working method based on a large data stream and batch integrated processing system, characterized in that: Executed based on the large data stream and batch integrated processing system according to any one of claims 1 to 6, including a stream data processing process and a batch data processing process; The stream data processing process includes: The backend management unit sends the data processing algorithm to the data warehouse and the computing unit, obtains the stream data to be processed through the data forwarding service unit, and transmits the stream data to be processed to the distributed message queue unit for queuing. At the same time, the backend management unit sends the data processing parameters corresponding to the stream data to be processed to the distributed message queue unit for queuing; The data warehouse and computing unit obtain the stream data to be processed, the data processing algorithm, and the data processing parameters. Combined with the data processing algorithm and data processing parameters, the Flink cluster in the data warehouse and computing unit analyzes and processes the stream data to be processed, obtaining the structured topic data corresponding to the stream data to be processed and the stream data analysis results. The structured subject data is stored in the Hbase cluster, and the stream data analysis results are transmitted to the MySQL database of the background management unit and the distributed storage unit respectively; The stream data analysis results are transmitted to the business application unit through the distributed storage unit, and the business application unit forwards the stream data analysis results to the client for visualization, completing the stream data processing; The batch data processing process includes: Send historical data and data processing algorithms to the data warehouse and computing unit through the background management unit, and send data processing parameters to the MySQL database through the background management unit; Use the Flink cluster to read historical data from the HBase cluster and the distributed file system HDFS, or obtain historical data issued by the background management unit through the Flink cluster. Combined with the data processing algorithm and the data processing parameters in the MySQL database of the background management unit, the historical data is processed and analyzed to obtain batch data analysis results, which are then stored in the distributed storage unit or MySQL database. The batch data analysis results are transmitted to the business application unit through the distributed storage unit, and the business application unit forwards the batch data analysis results to the client for visualization, or the batch data analysis results in the Mysql database are visualized through the background management unit to complete the batch data processing.

8. The working method according to claim 7, characterized in that: Analyze and process the stream data to be processed through the Flink cluster in the data warehouse and computing unit, including: When the Flink cluster starts working for the first time, the default data processing parameters in the MySQL database of the background management unit are read through the Flink cluster. Analyze and process the stream data to be processed according to the default data processing parameters; When data processing parameters need to be changed, the dynamic data processing parameters are sent to the distributed message queue unit through the background management unit, and the dynamic data processing parameters are forwarded to the data warehouse and computing unit through the distributed message queue unit to realize dynamic parameter configuration.

9. The working method according to claim 7, characterized in that: Also includes: Convert batch data into stream data, and use the Flink cluster to perform fusion analysis on the original stream data and the converted stream data.

Images