Data processing method, device, equipment and storage medium based on computing framework

Abstract

The present application provides a data processing method, device, equipment and storage medium based on a computing framework, wherein the method includes: when receiving data based on the computing framework, when determining to limit the data receiving speed, determining a first data volume, the first data volume is the first number of data items corresponding to the current time slice; when determining that the data difference between the first data volume and the pre-stored second data volume is greater than a preset value, and the first data volume is less than the second data volume, the second data volume is the second number of data items corresponding to the previous time slice, merging the data in the queue; and processing the merged data belonging to the same batch in a batch processing manner. The data in the queue is merged and processed to obtain at least one batch of merged data. By merging the data in the queue, the data can be processed in advance, avoiding continuous accumulation of data, and effectively improving the data processing efficiency based on the computing framework.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device and storage medium based on a computing framework. Background Art

[0002] With the development of data processing technology, the requirements for big data processing are getting higher and higher. Now, in order to meet the needs of real-time data processing, a Spark Streaming computing framework is provided. Spark Streaming can provide rich APIs and a high-speed execution engine based on memory.

[0003] Now, the Spark Streaming computing framework can be used to receive data at each time slice, and then process the data at each time slice as a whole. Specifically, the Spark Streaming computing framework specifies the length of the time slice, receives data in the first time slice, and then processes the data corresponding to the first time slice as a batch, then receives data in the second time slice, and then processes the data corresponding to the second time slice as a batch, and so on.

[0004] However, in the prior art, when receiving and processing data based on the Spark Streaming computing framework, since the data on each time slice is processed as a whole, it is necessary to first receive data within a time slice, and then process the data corresponding to the time slice. As a result, the data corresponding to the previous time slice is not processed yet, and the data of the next time slice begins to be received, which will cause data accumulation and low data processing efficiency. Summary of the invention

[0005] The present application provides a data processing method, device, equipment and storage medium based on a computing framework, which are used to solve the problems of data accumulation and low data processing efficiency in the data processing process based on the computing framework.

[0006] In a first aspect, the present application provides a data processing method based on a computing framework, the method comprising:

[0007] When receiving data based on the computing framework, when determining to limit the data receiving speed, determining a first data amount, wherein the first data amount is a first number of data items corresponding to the current time slice, and the data corresponding to the current time slice is data in a processing state;

[0008] When it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than a preset value, and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, the data in the queue are merged to obtain merged data belonging to the same batch, wherein the data in the queue are data received and not processed in the current time slice;

[0009] The merged data belonging to the same batch will be processed in batch processing mode.

[0010] Furthermore, merging the queued data to obtain merged data belonging to the same batch includes:

[0011] Determine the batch number N according to the total number of data items in the queue and the preset average number of data items, where N is a positive integer greater than or equal to 1;

[0012] According to the number of batches N, the data in the queue are merged to obtain N batches of merged data;

[0013] The merged data belonging to the same batch are processed in batch processing, including:

[0014] Each batch of N batches of merged data is processed in a batch processing manner.

[0015] Furthermore, the number of batches

[0016] Wherein, S is the total number of data items, and T is the average number of data items.

[0017] Furthermore, before determining the number N of batches according to the total number of data items in the queue and the preset average number of data items, the method further includes:

[0018] Obtaining the number of data items in a time slice of data corresponding to each time slice of a plurality of time slices, wherein the plurality of time slices are time slices before the current time slice;

[0019] The average value of the number of data items in each of the time slices is taken as the average number of data items.

[0020] Further, when determining to limit the data receiving speed, before determining the first data amount, the method further includes:

[0021] Acquire a first time length of the current time slice, and acquire a second time length required for processing data corresponding to the current time slice;

[0022] When it is determined that the second time length is greater than the first time length, the data receiving speed is limited.

[0023] Furthermore, the method further comprises:

[0024] When it is determined that the second time length is less than or equal to the first time length, the data corresponding to each time slice is processed at a preset speed.

[0025] Furthermore, the method further comprises:

[0026] When it is determined that the data difference between the first data amount and the pre-stored second data amount is less than or equal to a preset value, and the first data amount is less than the second data amount, or when it is determined that the first data amount is greater than or equal to the second data amount, the data corresponding to each time slice is processed at a preset speed.

[0027] In a second aspect, the present application provides a data processing device based on a computing framework, the device comprising:

[0028] A first determining unit, configured to determine a first data volume when determining to limit a data receiving speed when receiving data based on a computing framework, wherein the first data volume is a first number of data items corresponding to a current time slice, and the data corresponding to the current time slice is data in a processing state;

[0029] a merging unit, configured to merge the data in the queue to obtain merged data belonging to the same batch, when it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than a preset value and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, wherein the data in the queue is data received and not processed in the current time slice;

[0030] The first processing unit is used to process the merged data belonging to the same batch in a batch processing manner.

[0031] Furthermore, the merging unit comprises:

[0032] A determination module, used to determine the number of batches N according to the total number of data items in the queue and a preset average number of data items, where N is a positive integer greater than or equal to 1;

[0033] A merging module, used for merging the data in the queue according to the number of batches N, to obtain N batches of merged data;

[0034] The first processing unit is specifically configured to:

[0035] Each batch of N batches of merged data is processed in a batch processing manner.

[0036] Furthermore, the number of batches

[0037] Wherein, S is the total number of data items, and T is the average number of data items.

[0038] Furthermore, the device further comprises:

[0039] The second determination unit is used to obtain the number of data items in a time slice corresponding to each time slice of a plurality of time slices before the determination module determines the batch number N based on the total number of data items in the queue and a preset average number of data items, wherein the plurality of time slices are time slices before the current time slice; and take the average value of the number of data items in each of the time slices as the average number of data items.

[0040] Furthermore, the device further comprises:

[0041] an acquiring unit, configured to acquire, when the first determining unit determines to limit the data receiving speed, a first time length of the current time slice before determining the first data amount, and acquire a second time length required for processing data corresponding to the current time slice;

[0042] A rate limiting unit is used to limit the data receiving speed when it is determined that the second time length is greater than the first time length.

[0043] Furthermore, the device further comprises:

[0044] The second processing unit is configured to process data corresponding to each time slice at a preset speed when it is determined that the second time length is less than or equal to the first time length.

[0045] Furthermore, the device further comprises:

[0046] The third processing unit is used to process the data corresponding to each time slice at a preset speed when it is determined that the data difference between the first data amount and the pre-stored second data amount is less than or equal to a preset value and the first data amount is less than the second data amount, or when it is determined that the first data amount is greater than or equal to the second data amount.

[0047] In a third aspect, the present application provides a data processing device based on a computing framework, comprising units or means for executing each step of any method of the first aspect above.

[0048] In a fourth aspect, the present application provides a data processing device based on a computing framework, comprising a processor, a memory, and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement any method of the first aspect.

[0049] In a fifth aspect, the present application provides a data processing device based on a computing framework, comprising at least one processing element or chip for executing any of the methods of the first aspect above.

[0050] In a sixth aspect, the present application provides a computer program, which, when executed by a processor, is used to execute any one of the methods of the first aspect above.

[0051] In a seventh aspect, the present application provides a computer-readable storage medium on which the computer program of the sixth aspect is stored.

[0052] The present application provides a data processing method, apparatus, device and storage medium based on a computing framework. When receiving data based on the computing framework, when determining to limit the data receiving speed, a first data volume is determined, wherein the first data volume is the first number of data items corresponding to the current time slice, and the data corresponding to the current time slice is data in a processing state; when it is determined that the data difference between the first data volume and the pre-stored second data volume is greater than a preset value, and the first data volume is less than the second data volume, wherein the second data volume is the second number of data items corresponding to the previous time slice, the data in the queue are merged to obtain merged data belonging to the same batch, wherein the data in the queue are data received and not processed within the current time slice; and the merged data belonging to the same batch are processed in a batch processing manner. In the process of data processing based on the Spark Streaming computing framework, the amount of data received in the current time slice is much smaller than the amount of data received in the previous time slice, so it can be determined that the back pressure mechanism of Spark Streaming is effective. At this time, the amount of data received in each time slice after the current time slice starts is small; then, the data received in multiple time slices can be obtained, and these data are waiting to be processed, and these data constitute the data in the queue. Then, since the amount of data in each time slice after the back pressure mechanism takes effect is small, in order to avoid continuous data backlog, the data in the queue can be merged and processed to obtain at least one batch of merged data; the merged data of each batch is taken as a batch, and then the merged data of each batch is processed in batch processing. In this way, the data in the queue is merged and processed, and the data can be processed in advance, avoiding continuous data backlog and accumulation; improving the throughput of data processing, and effectively improving the data processing efficiency based on the Spark Streaming computing framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0055] Figure 2 A flowchart of a data processing method based on a computing framework provided in an embodiment of the present application;

[0056] Figure 3 A flowchart of another data processing method based on a computing framework provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of data processing based on Spark Steaming provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the architecture of the Spark Steaming computing framework provided in an embodiment of the present application;

[0059] Figure 6 The execution process of the back pressure mechanism based on the Spark Steaming computing framework provided in the embodiment of the present application;

[0060] Figure 7 A schematic diagram of the structure of a data processing device based on a computing framework provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of another data processing device based on a computing framework provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the structure of a data processing device based on a computing framework provided in an embodiment of the present application.

[0063] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0065] The embodiments of the present application can be applied to the field of data processing technology. The embodiments of the present application can be implemented by using a data processing apparatus based on a computing framework, or a data processing device based on a computing framework, or a terminal device, or a server, or other apparatus or equipment.

[0066] First, the terms involved in this application are explained:

[0067] 1) Data Warehouse (DW for short): It is a storage combination of data.

[0068] 2) Spark: It is a fast and general computing engine designed for large-scale data processing. It is better suited for algorithms that require iteration, such as data mining and machine learning.

[0069] 3) Spark Streaming: A framework for processing stream data built on Spark. It can divide the data stream into small time segments (unit: seconds) and process the data in each time segment in a manner similar to batch processing.

[0070] 4) Resilient Distributed Datasets (RDD): It is an abstract concept of distributed memory. RDD provides a highly restricted shared memory model.

[0071] 5) Partition: An RDD is physically divided into multiple partitions; these partitions can be distributed on different nodes. Partition is the basic processing unit of Spark computing tasks.

[0072] 6)Hadoop: It is a distributed system infrastructure; it mainly solves the distributed computing and storage problems of massive data.

[0073] The specific application scenarios of this application are: With the development of network technology, network data has been widely used in various fields. In addition, a large amount of data traffic will be generated on the Internet; and data usually has a large volatility. For example, in various activities or sudden hot events, the business traffic of data will increase dramatically in a short period of time, thus forming a huge traffic burr. Especially in the e-commerce industry, the above characteristics of data will be more obvious.

[0074] When the speed of data inflow is much higher than the speed of data processing, that is, when the data receiving speed is much higher than the data processing speed, it will put a huge load pressure on the data processing system; if the load pressure of the stream processing system cannot be well alleviated, it will lead to cluster resource exhaustion and even cluster crash. Therefore, it is necessary to ensure the stability of the data processing system.

[0075] Provides a batch processing framework MapReduce, which is a programming model that can be used for parallel computing of large-scale data sets. MapReduce can process data offline, but MapReduce cannot meet the needs of data with high real-time requirements. For example, MapReduce cannot process data in real-time data warehouses, real-time recommendations, user behavior analysis, and other scenarios in a timely manner.

[0076] Now, Spark is provided to facilitate distributed computing. Spark is a distributed computing framework similar to MapReduce. The core of Spark is elastic distributed data sets. However, Spark provides a richer model than MapReduce. Spark can quickly iterate data sets multiple times in memory. Therefore, based on the above features, Spark can support complex data mining algorithms, graph computing algorithms and other algorithms.

[0077] Based on Spark, Streaming was proposed. Spark Streaming is a real-time computing framework built on Spark. Spark Streaming expands Spark's ability to process large-scale streaming data. In addition, Spark Streaming can process data with high real-time requirements in a timely manner. Spark Streaming can provide a rich application programming interface (API), and Spark Streaming can complete data processing based on memory; furthermore, Spark Streaming combines streaming, batch processing, and user interaction to complete data query and data.

[0078] Based on the above introduction, we can see that the advantages of Spark Streaming are: it can run on hundreds of nodes and achieve a delay of seconds; Spark Streaming uses memory-based Spark as the execution engine, which makes it efficient and fault-tolerant; Spark Streaming integrates Spark's batch processing capabilities and interactive query capabilities; Spark Streamin provides a simple interface similar to batch processing, but can implement complex algorithms.

[0079] However, through the Spark Streaming computing framework, data is received in each time slice; then, the data in each time slice is processed as a whole. Specifically, through the Spark Streaming computing framework, the time length of the time slice is specified; data is received in the first time slice, and then the data corresponding to the first time slice is processed as a batch; then, data is received in the second time slice, and then the data corresponding to the second time slice is processed as a batch; and so on.

[0080] However, when receiving and processing data based on the Spark Streaming computing framework, since the data on each time slice is processed as a whole, it is necessary to first receive data within a time slice, and then process the data corresponding to the time slice. As a result, the data corresponding to the previous time slice has not been processed yet, and the data of the next time slice will start to be received, which will cause data accumulation and low data processing efficiency.

[0081] The data processing method, device, equipment and storage medium based on the computing framework provided in this application are intended to solve the above technical problems in the prior art.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application, such as Figure 1As shown, data can be obtained from various data sources and then input into Spark Streaming; Spark Streaming can process the data; then, the processed data is input into various data receiving sources. Among them, data sources include but are not limited to the following data sources: Kafka, Flume, HDFS (Hadoop Distributed File System, HDFS), Kinesis. Data receiving sources include but are not limited to the following data receiving sources: databases, dashboards. Among them, Kafka is a high-throughput distributed publish-subscribe messaging system; Flume is a highly available, highly reliable, distributed system for massive log collection, aggregation and transmission provided by Cloudera; HDFS is a highly fault-tolerant system that can provide high-throughput data access and is very suitable for applications on large-scale data sets; Kinesis is an AWS cloud service for collecting real-time streaming data.

[0084] Figure 2 A flowchart of a data processing method based on a computing framework provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0085] 101. When receiving data based on a computing framework, when determining to limit the data receiving speed, determine a first data volume, wherein the first data volume is the first number of data items corresponding to a current time slice, and the data corresponding to the current time slice is data in a processing state.

[0086] In this embodiment, the execution subject of this embodiment may be a data processing apparatus based on a computing framework, or a data processing device based on a computing framework, or a terminal device, or a server, or other apparatus or device.

[0087] In the process of data processing based on the Spark Steaming computing framework, since a back-pressure mechanism can be provided for Spark Steaming, when it is determined that the data receiving speed is much lower than the data processing speed, the data receiving speed can be reduced, thereby reducing the amount of data received based on the computing framework.

[0088] In the process of data processing based on the Spark Steaming computing framework, data can be received based on the computing framework, and data can be received in each time slice. After receiving the data, the data corresponding to each time slice is processed. During the processing, the next time slice starts again, and then data will be received in the next time slice. Therefore, each time slice has a corresponding data volume, and the data volume has the number of data items.

[0089] Based on the above process, after limiting the data receiving speed, data will still be received within the current time slice, and then the data corresponding to the current time slice can be obtained. Then, the number of data items corresponding to the current time slice can be determined. For ease of distinction, the number of data items corresponding to the current time slice is called the first number of data items; thus, the first data volume of the data corresponding to the current time slice can be obtained, and the first data volume is the first number of data items.

[0090] And, at this time, the data corresponding to the current time slice is received and the data corresponding to the current time slice is processed, that is, the data corresponding to the current time slice is in a processing state. At this time, the data corresponding to the current time slice is treated as a batch of data for batch processing.

[0091] For example, in the process of processing data based on Spark Steaming, data is received in time slice 1, and then data corresponding to time slice 1 is obtained, and then the data corresponding to time slice 1 is processed at a preset speed, that is, the data corresponding to time slice 1 is batch processed as a batch of data; data is received in time slice 2 after time slice 1, and then data corresponding to time slice 2 is obtained, and then the data corresponding to time slice 2 is processed at a preset speed, that is, the data corresponding to time slice 2 is batch processed as a batch of data; then the data receiving speed of time slice 3 after time slice 2 is limited based on the back pressure mechanism. At this time, the data corresponding to time slice 3 is received, and then the first data volume is obtained, which is the first number of data items corresponding to time slice 3; at this time, the data corresponding to the current time slice 3 is started to be processed, but the data corresponding to the current time slice 3 has not been processed yet.

[0092] 102. When it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than the preset value, and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, the data in the queue are merged to obtain merged data belonging to the same batch, wherein the data in the queue are data received and unprocessed within the current time slice.

[0093] In this embodiment, since the data of a time slice before the current time slice has been received, that is, the data of the previous time slice has been received, the second number of data items corresponding to the previous time slice can be determined, and thus, the second number of data items is used as the second data amount.

[0094] Then, the first data volume of step 101 is compared with the second data volume to determine whether the first data volume is smaller than the second data volume; when it is determined that the first data volume is smaller than the second data volume, the data difference is obtained by subtracting the first data volume from the second data volume; then, it is determined whether the data difference is greater than a preset value; when it is determined that the data difference is greater than the preset value, it is determined that the first data volume is much smaller than the second data volume. Thus, it can be determined that the amount of data received in the current time slice is much smaller than the amount of data received in the previous time slice, and further it can be determined that the data receiving speed is indeed reduced, that is, it is determined that the back pressure mechanism based on Spark Streaming is effective. For example, 200,000 data items are received in each batch normally, and 500 data items are received in each batch after the back pressure mechanism takes effect.

[0095] At this time, since the data corresponding to the current time slice has been completely received, the data corresponding to the current time slice is being processed in batches, but the data corresponding to the current time slice has not been completely processed, and the data in the next time slice has begun to be received, so the data in the next time slice enters the queue state, and the data in the next time slice waits for data processing. At this time, the data in the next time slice is treated as the same batch of data and waits for batch processing. In addition, in the process of processing the data corresponding to the current time slice, the data of the time slice after the next time slice may also be received, so that all of them enter the queue state and become the data in the queue; it can be seen that the data corresponding to each time slice in the queue is treated as a batch of data.

[0096] Since the data receiving speed has been limited, and it has been determined that the first data amount is much smaller than the second data amount, that is, the received data amount has been significantly reduced.

[0097] At this time, if the data in each time slice after the current time slice is processed as a batch of data, then the data in each time slice after the current time slice needs to continue to wait for processing, and in the process of processing the data corresponding to the current time slice, the data in the subsequent time slice 7 will continue to be received, so the data will continue to be backlogged. In this embodiment, the data in the queue can be merged, and then the data in the queue can be divided into at least one batch to obtain at least one batch of merged data, and the data in each annotated merged data belongs to the current batch; by merging data, the batches of data can be reduced.

[0098] For example, the data in the queue can be merged into one batch, or the data in the queue can be divided into multiple batches.

[0099] For example, in the process of data processing based on Spark Steaming, data is received in time slice 1, and then data corresponding to time slice 1 is obtained, and then the data corresponding to time slice 1 is processed at a preset speed, that is, the data corresponding to time slice 1 is processed as a batch of data; data is received in time slice 2 after time slice 1, and then data corresponding to time slice 2 is obtained, and then the data corresponding to time slice 2 is processed at a preset speed, that is, the data corresponding to time slice 2 is processed as a batch of data; then the data receiving speed of time slice 3 after time slice 2 is limited based on the back pressure mechanism; After receiving data in time slot 3, the data corresponding to time slot 3 is processed. During the processing, the following data receiving process of time slots will be performed. At this time, the data corresponding to time slot 3 has not been processed yet. Data will be received in time slot 4, in time slot 5, and in time slot 6, that is, data corresponding to time slot 4, data corresponding to time slot 5, and data corresponding to time slot 6 are obtained. It can be seen that after processing the data corresponding to time slot 3, the data in the queue is obtained, and the data in the queue includes data corresponding to time slot 4, data corresponding to time slot 5, and data corresponding to time slot 6. If the data corresponding to time slot 4 is processed as a batch of data, then the data corresponding to time slot 5 and the data corresponding to time slot 6 need to continue to wait for processing, and in the process of processing the data corresponding to time slot 4, data corresponding to time slot 7 will also be received, so the data will continue to be backlogged. At this time, the data corresponding to time slot 4, the data corresponding to time slot 5, and the data corresponding to time slot 6 can be merged and processed to obtain at least one batch of merged data, for example, to obtain two batches of merged data.

[0100] 103. The merged data belonging to the same batch shall be processed in batch processing mode.

[0101] In this embodiment, after step 102, at least one batch of merged data is obtained; then, the merged data belonging to the same batch is processed in a batch processing manner.

[0102] For example, two batches of merged data are obtained, namely, the merged data of batch A and the merged data of batch B. Then, the merged data of batch A can be processed in a batch processing manner; after the merged data of batch A is processed, the merged data of batch B can be processed in a batch processing manner. Alternatively, two threads can be called to process the merged data of batch A and the merged data of batch B respectively at the same time, for example, thread 1 is called to process the merged data of batch A, and thread 2 is called to process the merged data of batch B at the same time.

[0103] In this embodiment, when receiving data based on a computing framework, when determining to limit the data receiving speed, a first data volume is determined, wherein the first data volume is the first number of data items corresponding to the current time slice, and the data corresponding to the current time slice is data in a processing state; when it is determined that the data difference between the first data volume and the pre-stored second data volume is greater than a preset value, and the first data volume is less than the second data volume, wherein the second data volume is the second number of data items corresponding to the previous time slice, the data in the queue are merged to obtain merged data belonging to the same batch, wherein the data in the queue are data received and not processed within the current time slice; and the merged data belonging to the same batch are processed in a batch processing manner. In the process of data processing based on the Spark Streaming computing framework, the amount of data received in the current time slice is much smaller than the amount of data received in the previous time slice. It can be determined that the back pressure mechanism of Spark Streaming is effective. At this time, the amount of data received in each time slice after the current time slice starts is small. Then, the data received in multiple time slices can be obtained. These data are waiting to be processed and constitute the data in the queue. Then, since the amount of data in each time slice is small after the back pressure mechanism takes effect, in order to avoid continuous data backlog, the data in the queue can be merged and processed to obtain at least one batch of merged data; the merged data of each batch is taken as a batch, and then the merged data of each batch is processed in batch processing. In this way, the data in the queue can be merged and processed in advance, avoiding continuous data backlog and accumulation; improving the throughput of data processing, and effectively improving the data processing efficiency based on the Spark Streaming computing framework.

[0104] Figure 3 A flowchart of another data processing method based on a computing framework provided in an embodiment of the present application. Figure 3 As shown, the method includes:

[0105] 201. When receiving data based on a computing framework, obtain a first time length of a current time slice, and obtain a second time length required for processing data corresponding to the current time slice.

[0106] In this embodiment, the execution subject of this embodiment may be a data processing apparatus based on a computing framework, or a data processing device based on a computing framework, or a terminal device, or a server, or other apparatus or device.

[0107] In the process of data processing based on the Spark Steaming computing framework, when receiving the data stream, the data stream can be split into time slices, where the time slice can be at the second level; then, the data corresponding to each time slice is processed in a batch processing manner, that is, the data corresponding to each time slice is taken as each batch, and each batch of data is processed in turn.

[0108] For example, Figure 4 A data processing diagram based on Spark Steaming provided in the embodiment of the present application is as follows: Figure 4 As shown, the data stream is input into the computing framework based on Spark Steaming, and Spark Steaming is used to process the data stream; based on Spark Steaming, data can be received in time slice t1, that is, the data received in time length t1 is taken as a batch, and then the data received in time slice t1 is processed; in the process of processing the data received in time slice t1, data will also be received in time slice t2, that is, the data received in time length t2 is taken as a batch, and after processing the data received in time slice t1, the data received in time slice t2 is processed; and after receiving the data in time slice t2, data will continue to be received in time slice t3, and so on. After processing the data received in each time slice, the data processing result corresponding to each time slice can be obtained.

[0109] The data source will continuously generate data. Therefore, when processing data based on Spark Streaming, the data receiving speed can be the data generation speed of the data source; Spark Streaming can use Receiver-based data receivers, or; Spark Streaming can use direct receiving method (Direct Approach). After receiving the data in a time slice, the data in a time slice needs to be processed; during the processing, the next time slice starts again, and then, data will be received in the next time slice; and so on. When the time required to process data (batch processing time) is greater than the time interval between time slices (batch interval, also known as batch processing interval), that is, the time for processing each batch of data is longer than the batch processing interval of Spark Streaming, so more and more data is received and backlogged; at this time, the data processing speed cannot keep up, which leads to the beginning of data accumulation, and further even leads to out of memory (OOM) problems, which leads to the failure of the entire data processing process.

[0110] To avoid data backlog, you can limit the maximum amount of data that can be received per second. For example, in Spark Streaming, you can configure the parameter spark.streaming.receiver.maxRate for each Receiver-based data receiver to limit the maximum amount of data received per second; or, in Spark Streaming, when using the Direct Approach to receive data, you can configure the parameter spark.streaming.kafka.maxRatePerPartition to limit the maximum number of data read per Kafka partition in each job, thereby limiting the data reading speed and the maximum amount of data received per second.

[0111] However, the above method of limiting the maximum amount of data received per second requires pre-estimation of the data processing speed and data generation speed of the data cluster; and because it is necessary to configure parameters to limit the maximum amount of data received per second, in Spark Streaming, after modifying the parameters, it is necessary to restart the application based on Spark Streaming, which will lead to interruption of data processing and affect data processing efficiency; and when the data generation speed is high, the data cannot be processed in time, resulting in low cluster resource utilization. Therefore, a back pressure mechanism can be provided on the basis of Spark Streaming to solve the above problems. The back pressure mechanism adapts the cluster data processing capacity by dynamically controlling the data receiving rate, without the need to estimate the data processing speed and data generation speed.

[0112] Figure 5 The schematic diagram of the architecture of the Spark Steaming computing framework provided in the embodiment of the present application is as follows: Figure 5 As shown, Spark Steaming includes at least the Executor framework, Spark Streaming Driver, and SparkContext. Among them, the Executor framework includes at least the following components: Receiver and Block Manager. Spark Streaming Driver includes at least the following components: Receiver Tracker, Job Generator, and JobScheduler. Therefore, when the back pressure mechanism is started, the data receiving rate of the receiver can be dynamically adjusted according to the execution information of the data processing job fed back by the component JobScheduler. In addition, the property parameter "spark.streaming.backpressure.enabled" is used to control whether the back pressure mechanism is enabled; the default value of the property parameter is false, indicating that the back pressure mechanism is not enabled; when the property parameter is true, it indicates that the back pressure mechanism is enabled.

[0113] In the process of data processing based on the Spark Steaming computing framework, the back pressure function is first enabled by setting the parameter spark.streaming.backpressure.enabled to true; however, the back pressure mechanism is not enabled at this time, that is, the back pressure is not effective. Then, in the process of data reception and processing, when receiving data in the current time slice, the time length of the current time slice can be determined; the time length of the current time slice is called the first time length T1. In the process of processing the data corresponding to the current time slice, the time length required to process the data corresponding to the current time slice can be determined; the time length required to process the data corresponding to the current time slice is called the second time length T2.

[0114] 202. When it is determined that the second time length is greater than the first time length, limit the data receiving speed.

[0115] In this embodiment, after step 201 , the first time length T1 is compared with the second time length T2 .

[0116] When it is determined that the second time length T2 is greater than the first time length T1, it is determined that the time required for data processing is greater than the time required for data reception, and further, it is determined that the data processing speed is less than the data reception speed, and further, it is determined that the data processing capacity based on Spark Streaming is insufficient; at this time, it is determined that back pressure takes effect. When it is determined that back pressure takes effect, the data reception speed is limited.

[0117] In one example, Figure 6 The execution process of the back pressure mechanism based on the Spark Steaming computing framework provided in the embodiment of the present application is as follows: Figure 6As shown in the figure, the execution process of the back pressure mechanism is as follows: add a new component ReceiverRateController to the original architecture of Spark Streaming, and the component ReceiverRateController is responsible for listening to the "OnBatchCompleted" event; then, the component ReceiverRateController extracts the processing delay (processingDelay) information and the scheduling delay (scheduling Delay) information. Then, the component Estimator estimates the maximum processing speed based on this information, that is, based on this information, the data receiving rate can be dynamically controlled; the component Input Stream based on the receiver (Receiver) forwards the maximum processing speed to the component BlockGenerator through the components ReceiverTracker and ReceiverSupervisor. In this way, the speed of data receiving is limited.

[0118] 203. When determining to limit the data receiving speed, determine a first data volume, wherein the first data volume is a first number of data items corresponding to the current time slice, and the data corresponding to the current time slice is data in a processing state.

[0119] In this embodiment, after the back pressure mechanism takes effect, the data is processed in batches. Moreover, after the back pressure, after the data receiving speed is limited, data will still be received in the current time slice, and then the data corresponding to the current time slice can be obtained. Then, the number of data items corresponding to the current time slice can be determined. For the sake of distinction, the number of data items corresponding to the current time slice is called the first number of data items; thus, the first data volume of the data corresponding to the current time slice can be obtained, and the first data volume is the first number of data items.

[0120] And, at this time, the data corresponding to the current time slice is received and the data corresponding to the current time slice is processed, that is, the data corresponding to the current time slice is in a processing state. At this time, the data corresponding to the current time slice is treated as a batch of data for batch processing.

[0121] 204. Obtain the number of data items in a time slice corresponding to each time slice in a plurality of time slices, wherein the plurality of time slices are time slices before the current time slice; and take the average of the number of data items in each time slice as the average number of data items.

[0122] In this embodiment, step 204 may be performed before step 205, or may be performed during the execution of step 205. In step 204, a parameter needs to be determined, and the parameter is the average number of items.

[0123] The number of data items in the time slice corresponding to each time slice in multiple time slices is obtained in advance, and these time slices are all time slices before the current time slice; that is, for each time slice, the number of data items in a time slice is the sum of the number of data items received in the time slice; then the number of data items in the time slice of the data corresponding to each time slice is accumulated to obtain the sum of the number of items, and then the sum of the number of items is averaged to obtain the average number of items.

[0124] For example, during the execution of the back pressure mechanism, P time slices have been executed, data has been received in each of the P time slices, and the number of data items in each of the P time slices is Q, which can be the same or different; the number of data items in the P time slices is averaged to obtain the average number of items.

[0125] 205. When it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than the preset value, and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, the number of batches N is determined according to the total number of data items in the queue and the preset average number of data items, wherein N is a positive integer greater than or equal to 1; wherein the data in the queue is the data received and unprocessed in the current time slice.

[0126] In one example, the batch size S is the total number of data items, and T is the average number of data items.

[0127] In this embodiment, since the data of a time slice before the current time slice has been received, that is, the data of the previous time slice has been received, the second number of data items corresponding to the previous time slice can be determined, and thus, the second number of data items is used as the second data amount.

[0128] Then, the first data volume in step 203 is compared with the second data volume, see Figure 2 As described in step 102, when it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than the preset value, and the first data amount is less than the second data amount, it is determined that the data in the queue needs to be merged; the total number of data items S of the data in the queue can be obtained, and it can be known that the total number of data items S is the total number of data items in the queue; the total number of data items S is divided by the average number of items T to obtain the number of batches N.

[0129] In an example, when the total number of data items S cannot divide the average number of data items T, the rounding up formula can be used: Get the batch number N.

[0130] 206. According to the number of batches N, the data in the queue are merged to obtain N batches of merged data.

[0131] In this embodiment, after step 205, the number of batches N to be generated is obtained, and then the data in the queue is distributed into N batches of merged data. For example, the data in the queue is evenly divided into N batches of merged data.

[0132] 207. Process each batch of the N batches of merged data in a batch processing manner.

[0133] In this embodiment, after step 206, N batches of merged data are obtained, and data processing can be performed in a batch processing manner for each batch of merged data.

[0134] 208. When it is determined that the data difference between the first data amount and the pre-stored second data amount is less than or equal to the preset value and the first data amount is less than the second data amount, or when it is determined that the first data amount is greater than or equal to the second data amount, the data corresponding to each time slice is processed at a preset speed.

[0135] In this embodiment, after step 204, the first data amount in step 101 is compared with the second data amount to determine whether the first data amount is smaller than the second data amount.

[0136] When it is determined that the first data volume is greater than or equal to the second data volume, it is determined that the data volume received in the current time slice is not less than the data volume received in the previous time slice; or, when it is determined that the data difference obtained by subtracting the first data volume from the second data volume is less than or equal to the preset value, it can be determined that the data volume received in the current time slice is not much less than the data volume received in the previous time slice. In both cases, it can be considered that the data receiving speed is not reduced, that is, the back pressure mechanism based on Spark Streaming has no effect and the back pressure mechanism is not effective.

[0137] Then, the data corresponding to each time slice can be processed at the original preset speed; at this time, the data corresponding to each time slice is still regarded as a batch; each batch of data is processed in sequence according to the batch processing method.

[0138] 209. When it is determined that the second time length is less than or equal to the first time length, process data corresponding to each time slice at a preset speed.

[0139] In this embodiment, after step 201, when it is determined that the second time length T2 is less than or equal to the first time length T1, it is determined that the time required for data processing is less than or equal to the time required for data reception, and further, it is determined that the data processing speed is greater than or equal to the data reception speed, and further, it is determined that the data processing capacity based on Spark Streaming is sufficient; at this time, it is determined that back pressure is not effective, and there is no need to limit the data reception speed. Continue to receive data in each subsequent time slice.

[0140] Then, the data corresponding to each time slice can be processed at the original preset speed; at this time, the data corresponding to each time slice is still regarded as a batch; each batch of data is processed in sequence according to the batch processing method.

[0141] This embodiment, based on the above embodiment, is based on the back pressure mechanism of the Spark Streaming computing framework, and determines the number of batches of data that need to be merged by judging the amount of data in the time slice; then, the data in the queue is merged as multiple batches of data; the data can be processed in advance to avoid continuous backlog and accumulation of data; the throughput of data processing is improved, and the data processing efficiency based on the Spark Streaming computing framework is effectively improved. In addition, when it is determined that the back pressure mechanism is not to be started, the data corresponding to each time slice is processed at the original preset speed; at this time, the data corresponding to each time slice is still regarded as a batch.

[0142] Figure 7 A schematic diagram of a data processing device based on a computing framework provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the data processing device based on the computing framework of this embodiment may include:

[0143] The first determination unit 31 is used to determine the first data volume when receiving data based on the computing framework and determining to limit the data receiving speed, wherein the first data volume is the first number of data items corresponding to the current time slice, and the data corresponding to the current time slice is data in a processing state.

[0144] The merging unit 32 is used to merge the data in the queue to obtain merged data belonging to the same batch when it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than a preset value and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, wherein the data in the queue is data received and unprocessed in the current time slice.

[0145] The first processing unit 33 is used to process the merged data belonging to the same batch in a batch processing manner.

[0146] The computing framework-based data processing device of this embodiment can execute the computing framework-based data processing method provided by any of the above embodiments, and its implementation principles and technical effects are similar, which will not be repeated here.

[0147] Figure 8 A schematic diagram of the structure of another data processing device based on a computing framework provided in an embodiment of the present application, Figure 7 Based on the embodiment shown, Figure 8 As shown, the data processing device based on the computing framework of this embodiment, the merging unit 32, includes:

[0148] The determination module 321 is used to determine the batch number N according to the total number of data items in the queue and the preset average number of data items, where N is a positive integer greater than or equal to 1.

[0149] The merging module 322 is used to merge the data in the queue according to the number of batches N to obtain N batches of merged data.

[0150] The first processing unit 33 is specifically used to perform data processing on each batch of N batches of merged data in a batch processing manner.

[0151] In one example, the batch size Among them, S is the total number of data items and T is the average number of data items.

[0152] In one example, the device provided by this embodiment further includes:

[0153] The second determination unit 41 is used to obtain the number of data items in a time slice corresponding to each time slice of a plurality of time slices before the determination module 321 determines the batch number N based on the total number of data items in the queue and a preset average number of data items, wherein the plurality of time slices are time slices before the current time slice; and take the average value of the number of data items in each time slice as the average number of data items.

[0154] In one example, the device provided by this embodiment further includes:

[0155] The acquisition unit 42 is used for the first determination unit to acquire the first time length of the current time slice and the second time length required for processing the data corresponding to the current time slice before determining the first data amount when determining to limit the data receiving speed.

[0156] The speed limiting unit 43 is used to limit the data receiving speed when it is determined that the second time length is greater than the first time length.

[0157] In one example, the device provided by this embodiment further includes:

[0158] The second processing unit 44 is configured to process the data corresponding to each time slice at a preset speed when it is determined that the second time length is less than or equal to the first time length.

[0159] In one example, the device provided by this embodiment further includes:

[0160] The third processing unit 45 is used to process the data corresponding to each time slice at a preset speed when it is determined that the data difference between the first data amount and the pre-stored second data amount is less than or equal to a preset value and the first data amount is less than the second data amount, or when it is determined that the first data amount is greater than or equal to the second data amount.

[0161] The computing framework-based data processing device of this embodiment can execute the computing framework-based data processing method provided by any of the above embodiments, and its implementation principles and technical effects are similar, which will not be repeated here.

[0162] Fig. 9 A schematic diagram of a data processing device based on a computing framework provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the embodiment of the present application provides a data processing device based on a computing framework, which can be used to perform Figure 2-Figure 6 The actions or steps of the data processing method based on the computing framework in the illustrated embodiment specifically include: a processor 2701 , a memory 2702 and a communication interface 2703 .

[0163] The memory 2702 is used to store computer programs.

[0164] Processor 2701 is used to execute the computer program stored in memory 2702 to implement Figure 2-Figure 6 The actions of the data processing method based on the computing framework in the illustrated embodiment will not be described in detail.

[0165] Optionally, the data processing device based on the computing framework may further include a bus 2704. The processor 2701, the memory 2702, and the communication interface 2703 may be interconnected via the bus 2704; the bus 2704 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The above bus 2704 may be divided into an address bus, a data bus, and a control bus. For ease of representation, Fig. 9Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0166] In the embodiments of the present application, the above embodiments can refer to and learn from each other, and the same or similar steps and nouns will not be repeated one by one.

[0167] Alternatively, part or all of the above modules can also be implemented in a chip of the data processing device based on the computing framework in the form of an integrated circuit. And they can be implemented separately or integrated together. That is, the above modules can be configured as one or more integrated circuits that implement the above methods, such as: one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA).

[0168] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 2702 including instructions, and the instructions can be executed by the processor 2701 of the data processing device based on the computing framework to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0169] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a data processing device based on a computing framework, enables the data processing device based on the computing framework to perform the above-mentioned data processing method based on the computing framework.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a terminal device or a data center by wired (for example, coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode to another website site, computer, terminal device or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a terminal device, a data center, etc. that contains one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0171] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0172] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method based on a computing framework, characterized in that: The method comprises: When receiving data based on the computing framework, when determining to limit the data receiving speed, determining a first data amount, wherein the first data amount is a first number of data items corresponding to the current time slice, and the data corresponding to the current time slice is data in a processing state; When it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than a preset value, and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, the data in the queue are merged to obtain merged data belonging to the same batch, wherein the data in the queue are data received and not processed in the current time slice; The merged data belonging to the same batch will be processed in batch processing mode.

2. The method according to claim 1, characterized in that The step of merging the queued data to obtain merged data belonging to the same batch includes: Determine the batch number N according to the total number of data items in the queue and the preset average number of data items, where N is a positive integer greater than or equal to 1; According to the number of batches N, the data in the queue are merged to obtain N batches of merged data; The merged data belonging to the same batch are processed in batch processing, including: Each batch of N batches of merged data is processed in a batch processing manner.

3. The method according to claim 2, characterized in that The number of batches Wherein, S is the total number of data items, and T is the average number of data items.

4. The method according to claim 2, characterized in that: Before determining the batch number N according to the total number of data items in the queue and the preset average number of data items, the method further includes: Obtaining the number of data items in a time slice of data corresponding to each time slice of a plurality of time slices, wherein the plurality of time slices are time slices before the current time slice; The average value of the number of data items in each of the time slices is taken as the average number of data items.

5. The method according to any one of claims 1 to 4, characterized in that: When determining to limit the data receiving speed, before determining the first data amount, the method further includes: Acquire a first time length of the current time slice, and acquire a second time length required for processing data corresponding to the current time slice; When it is determined that the second time length is greater than the first time length, the data receiving speed is limited.

6. The method according to claim 5, characterized in that The method further comprises: When it is determined that the second time length is less than or equal to the first time length, the data corresponding to each time slice is processed at a preset speed.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: When it is determined that the data difference between the first data amount and the pre-stored second data amount is less than or equal to a preset value, and the first data amount is less than the second data amount, or when it is determined that the first data amount is greater than or equal to the second data amount, the data corresponding to each time slice is processed at a preset speed.

8. A data processing device based on a computing framework, characterized in that: The device comprises: A first determining unit, configured to determine a first data volume when determining to limit a data receiving speed when receiving data based on a computing framework, wherein the first data volume is a first number of data items corresponding to a current time slice, and the data corresponding to the current time slice is data in a processing state; a merging unit, configured to merge the data in the queue to obtain merged data belonging to the same batch, when it is determined that the data difference between the first data amount and the pre-stored second data amount is greater than a preset value and the first data amount is less than the second data amount, wherein the second data amount is the second number of data items corresponding to the previous time slice, wherein the data in the queue is data received and not processed in the current time slice; The first processing unit is used to process the merged data belonging to the same batch in a batch processing manner.

9. The device according to claim 8, characterized in that The merging unit comprises: A determination module, used to determine the number of batches N according to the total number of data items in the queue and a preset average number of data items, where N is a positive integer greater than or equal to 1; A merging module, used for merging the data in the queue according to the number of batches N, to obtain N batches of merged data; The first processing unit is specifically configured to: Each batch of N batches of merged data is processed in a batch processing manner.

10. The device according to claim 9, characterized in that The number of batches Wherein, S is the total number of data items, and T is the average number of data items.

11. The device according to claim 9, characterized in that The device further comprises: The second determination unit is used to obtain the number of data items in a time slice corresponding to each time slice of a plurality of time slices before the determination module determines the batch number N based on the total number of data items in the queue and a preset average number of data items, wherein the plurality of time slices are time slices before the current time slice; and take the average value of the number of data items in each of the time slices as the average number of data items.

12. The device according to any one of claims 8 to 11, characterized in that: The device further comprises: an acquiring unit, configured to acquire, when the first determining unit determines to limit the data receiving speed, a first time length of the current time slice before determining the first data amount, and acquire a second time length required for processing data corresponding to the current time slice; A rate limiting unit is used to limit the data receiving speed when it is determined that the second time length is greater than the first time length.

13. The device according to claim 12, characterized in that The device further comprises: The second processing unit is configured to process data corresponding to each time slice at a preset speed when it is determined that the second time length is less than or equal to the first time length.

14. The device according to any one of claims 8 to 11, characterized in that: The device further comprises: The third processing unit is used to process the data corresponding to each time slice at a preset speed when it is determined that the data difference between the first data amount and the pre-stored second data amount is less than or equal to a preset value and the first data amount is less than the second data amount, or when it is determined that the first data amount is greater than or equal to the second data amount.

15. A data processing device based on a computing framework, characterized in that: include: processor, memory and computer program; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images