Data synchronization method and system based on Spark Streaming

Abstract

The present application relates to a data synchronization method and system based on Spark Streaming, wherein the data synchronization method includes: an incremental data acquisition step, which is used to acquire HBase incremental data based on Spark Streaming and store it in a discrete data stream DStream; a data regularization step, which is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and regularization rules of HBase to obtain regularized data; a data writing step, which is used to write the regularized data to the corresponding directory of HDFS to achieve data synchronization. Through the data synchronization method and system of the present application, data is incrementally acquired based on the Spark Streaming data stream, the amount of data processed is small, and the amount of access to the HBase cluster is small; at the same time, the Spark Streaming data stream is updated in quasi-real time, the processing speed is fast, and the time consumption is short, which realizes quasi-real-time incremental synchronization of the HBase database, reduces the amount of data during the synchronization process and improves work efficiency.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data synchronization method and system based on Spark Streaming. Background Art

[0002] With the development of big data technology, more and more companies are starting to build big data platforms for data processing. HBase database (Hadoop Database) is a highly reliable, high-performance, column-oriented, scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC server. Data in the HBase database can be synchronized through ETL (Extract-Transform-Load). However, how to quickly ETL and synchronize the data stored in the HBase database to the data warehouse Hive is becoming a problem faced by many companies.

[0003] The existing technology uses Phoenix to export data from HBase to HDFS, then performs ETL through MapReduce or Spark, and finally loads it into Hive. Or directly read data from HBase through MapReduce or Spark and perform ETL, output the results to HDFS (Hadoop Distributed File System), and finally load it into Hive, and use Hive to analyze and manage the data on HDFS. The existing technical solution is to read all the data summarized by Hbase every time. The amount of data is too large, which increases the burden on the cluster and takes a long time, so it can only be synchronized offline. Therefore, when faced with massive data, both MapReduce tasks and Spark tasks have high latency, and scheduled tasks are required to achieve the effect of automated processes.

[0004] An effective solution for quickly synchronizing data in an HBase database to Hive has not yet been proposed in the prior art. Summary of the invention

[0005] The embodiment of the present application provides a data synchronization method and system based on Spark Streaming. Spark Streaming is an extension of the Spark core interface. Spark Streaming can realize high-throughput, real-time streaming data processing with a fault-tolerant mechanism. The present application uses Spark Streaming to realize quasi-real-time incremental synchronization of the HBase database, reduce the amount of data during the synchronization process and improve work efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data synchronization method based on Spark Streaming, comprising:

[0007] The incremental data acquisition step is used to acquire HBase incremental data based on Spark Streaming and store it in a discrete data stream DStream;

[0008] A data normalization step is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and normalization rules of HBase to obtain normalized data;

[0009] The data writing step is used to write the regularized data into the corresponding directory of HDFS to achieve data synchronization.

[0010] Through the above steps, the embodiment of the present application uses the Spark Streaming data stream to incrementally acquire data, processes a small amount of data, and has a small amount of access to the HBase cluster; at the same time, the Spark Streaming data stream is updated in quasi-real time, with a fast processing speed and a short time consumption.

[0011] In some embodiments, the incremental data acquisition step further comprises:

[0012] A data receiver creation step is used to create a data receiver HBaseReceiver for querying HBase data class, wherein the HBaseReceiver inherits from the Receiver class of Spark;

[0013] The data acquisition step is used to acquire incremental data and store the incremental data in the discrete data stream DStream.

[0014] Since Spark Streaming does not have a Stream interface for obtaining HBase, the above steps create a custom data receiver HBaseReceiver, obtain HBase incremental data through the data receiver HBaseReceiver and store the data as DStream, thereby realizing data acquisition in the form of Spark Streaming data streams.

[0015] In some embodiments, the data writing step further comprises:

[0016] The single directory data output step is used to write the formatted data into the corresponding directory by creating a data output class AppendTextOutputFormat;

[0017] The multi-directory data output step is used to write the regularized data into corresponding multiple directories by creating a multi-directory data output class MultipleAppendTextOutputFormat. The structure of the directory is the partition table structure of Hive, and the data on HDFS is analyzed and managed through Hive;

[0018] Among them, the data output class AppendTextOutputFormat inherits from Spark's TextOutputFormat class, and the multi-directory data output class MultipleAppendTextOutputFormat inherits from Spark's MultipleOutputFormat. The above steps realize appending data based on existing files, avoiding the use of the TextOutputFormat class to generate a large number of files, and at the same time, the multi-directory data append writing is realized through the multi-directory data output class MultipleAppendTextOutputFormat.

[0019] Considering that in the incremental data acquisition step, if the amount of new data between the current acquisition time point and the previous acquisition time point is too large, there may be a situation where the acquisition operation cannot be completed within the set time period. In some embodiments, the incremental data acquisition step obtains HBase to obtain incremental data based on a job flow limiting rule. The job flow limiting rule sets a maximum number of extractions. If the current number of data to be extracted is greater than the maximum number of extractions, the data exceeding the maximum number of extractions is delayed for a set time before being extracted. The maximum number of extractions in the job flow limiting rule can be set by comprehensively considering the cluster processing capacity and the data acquisition time interval, and the impact of data peak flow on the system can be effectively reduced by delaying the extraction of data.

[0020] Spark provides a set of window operations, which perform statistical analysis on incremental updates of large-scale data through sliding window technology. Based on this, in some embodiments, in the incremental data acquisition step, the start time interval of the data receiver HBaseReceiver and the window sliding time interval of Spark are set, wherein the start time interval is smaller than the window sliding time interval. By way of example and not limitation, the window sliding time interval is 10 seconds, and the start time interval is 5 seconds.

[0021] In a second aspect, an embodiment of the present application provides a data synchronization system based on Spark Streaming, including:

[0022] The incremental data acquisition module is used to obtain HBase incremental data based on Spark Streaming and store it in a discrete data stream DStream;

[0023] A data normalization module is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and normalization rules of HBase to obtain normalized data;

[0024] The data writing module is used to write the regularized data into the corresponding directory of HDFS to achieve data synchronization.

[0025] In some embodiments, the incremental data acquisition module further includes:

[0026] Create a data receiver module, used to create a data receiver HBaseReceiver for querying HBase data class, wherein the data receiver HBaseReceiver inherits from the Receiver class of Spark;

[0027] The data acquisition module is used to acquire incremental data and store the incremental data in the discrete data stream DStream.

[0028] In some embodiments, the data writing module further comprises:

[0029] The single directory data output module is used to write the formatted data into the corresponding directory by creating a data output class AppendTextOutputFormat;

[0030] The multi-directory data output module is used to write the regularized data into corresponding multiple directories by creating a multi-directory data output class MultipleAppendTextOutputFormat. The structure of the directory is the partition table structure of Hive, and the data on HDFS is analyzed and managed through Hive;

[0031] The data output class AppendTextOutputFormat inherits from Spark's TextOutputFormat class, and the multi-directory data output class MultipleAppendTextOutputFormat inherits from Spark's MultipleOutputFormat. The above module implements appending data based on existing files, avoiding the use of the TextOutputFormat class to generate a large number of files, and at the same time implements multi-directory data appending writing through the multi-directory data output class MultipleAppendTextOutputFormat.

[0032] In some embodiments, the incremental data acquisition module acquires incremental data from HBase based on a job flow limiting rule, and the job flow limiting rule sets a maximum number of extraction items. If the number of data items currently to be extracted is greater than the maximum number of extraction items, the data exceeding the maximum number of extraction items will be delayed for a set time before being extracted.

[0033] In some embodiments, in the incremental data acquisition module, the start time interval of the data receiver HBaseReceiver and the window sliding time interval of Spark are set, wherein the start time interval is smaller than the window sliding time interval. By way of example but not limitation, the window sliding time interval is 10 seconds and the start time interval is 5 seconds.

[0034] Compared with the related art, the data synchronization method and system based on Spark Streaming provided in the embodiments of the present application use Spark Streaming to extract incremental data from HBase in the form of data stream in quasi-real time, with good real-time performance. Spark Streaming can divide the input data into segments of discrete data streams DStream according to the batch size. These discrete data streams can be seamlessly integrated with Spark components such as MLlib and Spark SQL.

[0035] The data in Hbase is incrementally acquired in the form of data stream based on Spark Streaming. On the one hand, only incremental data is acquired each time, and the data volume is small, so the access volume to the Hbase cluster is not large, which will not cause pressure on the Hbase cluster. On the other hand, Spark Streaming's quasi-real-time data stream has a fast processing speed and a relatively short processing time.

[0036] At the same time, this application normalizes the field names and field values ​​according to the field mapping relationship file to facilitate data writing to the corresponding partitions.

[0037] In summary, this solution achieves the effect of quasi-real-time incremental synchronization of data from Hbase to Hive.

[0038] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1is a flow chart of a data synchronization method based on Spark Streaming according to an embodiment of the present application;

[0041] Figure 2 It is a structural block diagram of a data synchronization system based on Spark Streaming according to an embodiment of the present application;

[0042] Figure 3 It is a data flow diagram of a data synchronization method based on Spark Streaming according to a preferred embodiment of the present application. Description of the drawings:

[0044] 0. Data synchronization system; 1. Incremental data acquisition module; 2. Data normalization module;

[0045] 3. Data writing module;

[0046] 11. Create data receiver module; 12. Data acquisition module;

[0047] 31. Single directory data output module; 32. Multi-directory data output module. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0049] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0050] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] This embodiment provides a data synchronization method based on Spark Streaming. Figure 1 is a flowchart of a data synchronization method based on Spark Streaming according to an embodiment of the present application, with reference to Figure 1 As shown, the process includes the following steps:

[0053] Incremental data acquisition step S1, for acquiring HBase incremental data based on Spark Streaming and storing it in a discrete data stream DStream;

[0054] The data normalization step S2 is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and normalization rules of HBase to obtain normalized data, wherein the field mapping relationship is the corresponding relationship between the field names before and after field normalization;

[0055] The data writing step S3 is used to write the regularized data into the corresponding directory of HDFS to achieve data synchronization.

[0056] The incremental data acquisition step S1 further includes:

[0057] A data receiver creation step S101 is used to create a data receiver HBaseReceiver for querying HBase data class, and the data receiver HBaseReceiver inherits from the Receiver class of Spark;

[0058] The data acquisition step S102 is used to acquire incremental data through a private method receive() and store the incremental data into a discrete data stream DStream by calling store(b.toString()).

[0059] The data writing step S3 further includes:

[0060] Single directory data output step S301, for writing the regularized data into the corresponding directory by creating a data output class AppendTextOutputFormat;

[0061] The multi-directory data output step S302 is used to write the regularized data into corresponding multiple directories by creating a multi-directory data output class MultipleAppendTextOutputFormat. The structure of the directory is the partition table structure of Hive, so as to analyze and manage the data on HDFS through Hive;

[0062] Among them, the data output class AppendTextOutputFormat inherits from Spark's TextOutputFormat class, and the multi-directory data output class MultipleAppendTextOutputFormat inherits from Spark's MultipleOutputFormat. The above steps implement appending data based on existing files, avoiding the use of the TextOutputFormat class to generate a large number of files, and at the same time, the multi-directory data output class MultipleAppendTextOutputFormat implements multi-directory data append writing.

[0063] Through the above steps, the embodiment of the present application uses the Spark Streaming data stream form to incrementally acquire data, which processes a small amount of data and has a small amount of access to the HBase cluster. At the same time, the Spark Streaming data stream is updated in quasi-real time, with fast processing speed and short time consumption.

[0064] In addition, since Spark Streaming does not provide a Stream interface for obtaining HBase, the above steps are used to create a custom data receiver HBaseReceiver, obtain HBase incremental data through the data receiver HBaseReceiver and store the data as DStream, thus realizing data acquisition based on the Spark Streaming data stream.

[0065] Considering that in the incremental data acquisition step S1, if the amount of new data between the current acquisition time point and the previous acquisition time point is too large, there may be a situation where the acquisition operation cannot be completed within the set time period. In some embodiments, the incremental data acquisition step S1 acquires incremental data from HBase based on a job flow limiting rule. The job flow limiting rule sets a maximum number of extractions. If the current number of data to be extracted is greater than the maximum number of extractions, the data exceeding the maximum number of extractions will be delayed for a set time before being extracted. The maximum number of extractions in the job flow limiting rule can be set by comprehensively considering the cluster processing capacity and the data acquisition time interval, and the impact of data peak flow on the system can be effectively reduced by delaying the extraction of data.

[0066] Spark provides a set of window operations, which perform statistical analysis on incremental updates of large-scale data through sliding window technology. Based on this, in some embodiments, in the incremental data acquisition step S1, the start time interval of the data receiver HBaseReceiver and the window sliding time interval of Spark are set, wherein the start time interval is smaller than the window sliding time interval. By way of example and not limitation, the window sliding time interval is 10 seconds and the start time interval is 5 seconds.

[0067] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0068] Figure 3 is a data flow diagram of a data synchronization method according to a preferred embodiment of the present application, with reference to Figure 3 shown.

[0069] First, execute the incremental data acquisition step S1, query the HBase data class through the custom created data receiver HBaseReceiver, the data receiver HBaseReceiver inherits from the Spark Receiver class, and obtains the HBase incremental data through the private method receive() and calls store(b.toString()) to write the data to DStream; set the data receiver HBaseReceiver startup frequency to once every 5 seconds, and Spark's window sliding is processed every 10 seconds.

[0070] Then, the data normalization step S2 is executed to perform ETL on the data. According to the Hbase field mapping relationship and the corresponding normalization rules, ETL processing is performed on the stream data in the discrete data stream DStream. The ETL is based on the Hive architecture.

[0071] Finally, execute the data writing step S3 to write the regularized data, that is, the data of the corresponding partition filtered out, to the corresponding directory of HDFS. Since the native TextOutputFormat will generate a large number of small files, the custom created AppendTextOutputFormat inherits TextOutputFormat and appends to the existing files; but in order to avoid writing all files to the given directory, the custom created MultipleAppendTextOutputFormat inherits MultipleOutputFormat to implement multi-directory append writing, and the directory structure follows the structure of the Hive partition table.

[0072] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] This embodiment also provides a data synchronization system based on Spark Streaming. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and conceivable.

[0074] Figure 2 is a structural block diagram of a data synchronization system based on Spark Streaming according to an embodiment of the present application, refer to Figure 2 As shown, the system includes:

[0075] Incremental data acquisition module 1, used to acquire HBase incremental data based on Spark Streaming and store it into a discrete data stream DStream;

[0076] Data normalization module 2 is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and normalization rules of HBase to obtain normalized data, wherein the field mapping relationship is the corresponding relationship between the field names before and after field normalization;

[0077] The data writing module 3 is used to write the regularized data into the corresponding directory of HDFS to achieve data synchronization.

[0078] The incremental data acquisition module 1 further includes:

[0079] Create a data receiver module 11, which is used to create a data receiver HBaseReceiver for querying HBase data class, and the data receiver HBaseReceiver inherits from the Receiver class of Spark;

[0080] The data acquisition module 12 is used to acquire incremental data through a private method receive() and store the incremental data into a discrete data stream DStream by calling store(b.toString()).

[0081] The data writing module 3 further comprises:

[0082] The single directory data output module 31 is used to write the regularized data into the corresponding directory by creating a data output class AppendTextOutputFormat;

[0083] The multi-directory data output module 32 is used to write the regularized data into corresponding multiple directories by creating a multi-directory data output class MultipleAppendTextOutputFormat. The structure of the directory is the partition table structure of Hive, so as to analyze and manage the data on HDFS through Hive;

[0084] Among them, the data output class AppendTextOutputFormat inherits from Spark's TextOutputFormat class, and the multi-directory data output class MultipleAppendTextOutputFormat inherits from Spark's MultipleOutputFormat. The above module implements appending data based on existing files to avoid using the TextOutputFormat class to generate a large number of files, and at the same time implements multi-directory data appending writing through the multi-directory data output class MultipleAppendTextOutputFormat.

[0085] Based on the above modules, the embodiment of the present application uses Spark Streaming data stream to incrementally acquire data, processes a small amount of data, and has a small amount of access to the HBase cluster; at the same time, the Spark Streaming data stream is updated in quasi-real time, with fast processing speed and short time consumption.

[0086] In some of the embodiments, the incremental data acquisition module 1 acquires incremental data from HBase based on a job flow limiting rule. The job flow limiting rule sets a maximum number of extraction items. If the number of data items currently to be extracted is greater than the maximum number of extraction items, the data exceeding the maximum number of extraction items will be delayed for a set time before being extracted. By delaying the extraction of data, the impact of data peak flow on the system is effectively reduced.

[0087] In some of the embodiments, in the incremental data acquisition module 1, the start time interval of the data receiver HBaseReceiver and the window sliding time interval of Spark are set, wherein the start time interval is smaller than the window sliding time interval. By way of example but not limitation, the window sliding time interval is 10 seconds and the start time interval is 5 seconds.

[0088] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0089] The Spark Streaming-based data synchronization method and system provided in this embodiment use Spark Streaming to extract incremental data from HBase in quasi-real time in the form of data streams, with good real-time performance, and perform ETL according to field regularization rules. Spark Streaming can divide input data into discrete data streams DStream according to batch size, and these discrete data streams can be seamlessly integrated with Spark components such as MLlib and Spark SQL.

[0090] The data stream based on Spark Streaming is used to incrementally obtain data from Hbase. On the one hand, only incremental data is obtained each time, the amount of data processed is small, and the number of visits to the Hbase cluster is not large, which will not cause pressure on the Hbase cluster; on the other hand, Spark Streaming's quasi-real-time data stream has a fast processing speed and a relatively short processing time.

[0091] At the same time, this application normalizes the field names and field values ​​according to the field mapping relationship file to facilitate data writing to the corresponding partitions.

[0092] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method based on Spark Streaming, characterized in that: include: The incremental data acquisition step is used to acquire HBase incremental data based on Spark Streaming and store it in a discrete data stream DStream; A data normalization step is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and normalization rules of HBase to obtain normalized data; The data writing step is used to write the regularized data into the corresponding directory of HDFS to achieve data synchronization. The data writing step further includes: A single directory data output step is used to write the regularized data into the corresponding directory by creating a data output class; The multi-directory data output step is used to write the regularized data into the corresponding multiple directories by creating a multi-directory data output class. The data output class AppendTextOutputFormat inherits from the TextOutputFormat class of Spark, and the multi-directory data output class MultipleAppendTextOutputFormat inherits from the MultipleOutputFormat of Spark. The incremental data acquisition step further comprises: Create a data receiver step, used to create a data receiver for querying HBase data class; A data acquisition step is used to acquire incremental data and store the incremental data in the discrete data stream DStream; The incremental data acquisition step acquires incremental data from HBase based on a job flow limiting rule, and the job flow limiting rule sets a maximum number of extracted items. If the number of data items to be extracted currently is greater than the maximum number of extracted items, the data exceeding the maximum number of extracted items will be delayed for a set time before being extracted; In the incremental data acquisition step, a startup time interval of the data receiver and a window sliding time interval of Spark are set, wherein the startup time interval is smaller than the window sliding time interval.

2. A data synchronization system based on Spark Streaming, characterized in that: include: The incremental data acquisition module is used to obtain HBase incremental data based on Spark Streaming and store it in a discrete data stream DStream; A data normalization module is used to process the data in the discrete data stream DStream through ETL according to the field mapping relationship and normalization rules of HBase to obtain normalized data; A data writing module is used to write the regularized data into the corresponding directory of HDFS to achieve data synchronization. The data writing module further includes: A single directory data output module is used to write the regularized data into the corresponding directory by creating a data output class; The multi-directory data output module is used to write the regularized data into the corresponding multiple directories by creating a multi-directory data output class. The data output class AppendTextOutputFormat inherits from the Spark TextOutputFormat class, and the multi-directory data output class MultipleAppendTextOutputFormat inherits from the Spark MultipleOutputFormat. The incremental data acquisition module further includes: Create a data receiver module to create a data receiver for querying HBase data class; A data acquisition module, used to acquire incremental data and store the incremental data in the discrete data stream DStream; The incremental data acquisition module acquires incremental data from HBase based on a job flow limiting rule. The job flow limiting rule sets a maximum number of extracted items. If the number of data items to be extracted currently is greater than the maximum number of extracted items, the data exceeding the maximum number of extracted items will be delayed for a set time before being extracted. In the incremental data acquisition module, the startup time interval of the data receiver and the window sliding time interval of Spark are set, wherein the startup time interval is smaller than the window sliding time interval.

Images