Big data real-time processing method for mass data streams

Abstract

The invention relates to the technical field of big data, in particular to a big data real-time processing method for mass data streams, which comprises the following steps of: firstly obtaining a state parameter of each processing device in a cluster to obtain a processing pressure value of each processing device, then obtaining a to-be-processed data stream in a preset time period, and then according to the type and sequence of the to-be-processed data, obtaining a processing pressure value of each processing device; according to the method, a processing priority is established for each piece of to-be-processed data, finally, the to-be-processed data in the to-be-processed data stream is allocated to a plurality of processing devices for processing with the best real-time performance, and the real-time performance is obtained according to the processing pressure value and the processing priority. Compared with the prior art, the method has the advantages that the current processing capacity of each device in the cluster is quantized into the processing pressure value, and then the processing priority is established in combination with the priority and sequence of the data to be processed in the data stream, so that each data can be distributed to the appropriate processing device for processing according to the processing pressure value and the processing priority; and the real-time performance of data processing is ensured.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a real-time big data processing method for massive data streams. Background Technology

[0002] In today's digital age, the generation of massive amounts of data has become the norm. Businesses, organizations, and even individuals face enormous challenges in processing this vast amount of data. Real-time processing of big data requires powerful computing capabilities.

[0003] Traditional processing methods can no longer meet the demands of real-time processing, as big data real-time processing requires data extraction, transformation, and loading to be completed within milliseconds. Traditional database technologies are inadequate because they typically face problems such as excessively large data volumes and slow read / write speeds. Therefore, high-performance distributed cluster technologies, such as Hadoop and Spark, have been designed to enable rapid data storage, indexing, and retrieval, supporting the processing and analysis of massive amounts of data.

[0004] While many powerful distributed cluster computing platforms exist, high-concurrency data access and complex data streams can lead to data processing latency, making it challenging to ensure real-time data processing. Therefore, there is a particular need for a real-time big data processing method for massive data streams. Summary of the Invention

[0005] In view of this, it is necessary to provide a real-time big data processing method for massive data streams to solve the problem of how to ensure real-time performance when processing massive data streams.

[0006] This invention provides a method for real-time big data processing of massive data streams, comprising:

[0007] Obtain the status parameters of each processing device in the cluster, and obtain the processing pressure value of each processing device based on the status parameters;

[0008] Acquire the data stream to be processed within a preset time period;

[0009] Based on the type and order of the data to be processed in the data stream to be processed, a processing priority is established for each data to be processed.

[0010] The data to be processed in the data stream is distributed to multiple processing devices for processing with optimal real-time performance, wherein the real-time performance is obtained based on the processing pressure value and the processing priority.

[0011] Further, the step of establishing a processing priority for each piece of data to be processed based on the type and order of the data to be processed in the data stream includes:

[0012] The type of data to be processed is determined based on the message protocol type of the data to be processed, wherein the types of data to be processed include reliable data and unreliable data;

[0013] Based on the type of data to be processed, an initial priority is assigned to each type of data.

[0014] Based on the sequential position of each piece of data to be processed in the data stream, the initial priority of each piece of data to be processed is optimized to obtain the processing priority of each piece of data to be processed.

[0015] Furthermore, the processing priority is obtained according to the following formula:

[0016]

[0017] P i ∈[0,1]

[0018] P′ i ∈[0,1]

[0019] Among them, P i P′ represents the processing priority of the i-th data in the data stream to be processed. A larger value indicates a higher priority. i The initial priority of the i-th data to be processed in the data stream to be processed is given by k, where a larger value indicates a higher priority. k is the first adjustment parameter, i is the position of the data to be processed in the data stream to be processed, and n is the total number of data to be processed in the data stream to be processed.

[0020] Furthermore, the real-time performance is obtained according to the following formula:

[0021]

[0022] S j ∈[0,1]

[0023] Among them, RT ij To ensure the real-time performance of allocating the i-th data item in the data stream to the j-th processing device in the cluster, a larger value of S indicates better real-time data processing. j α represents the processing pressure value of the j-th processing device in the cluster. The larger the value, the greater the processing pressure of the processing device. j is the number of the processing device in the cluster, and α is the second adjustment parameter.

[0024] Further, the status parameters include device hardware parameters and device operating condition parameters; the step of obtaining the status parameters of each processing device in the cluster, and obtaining the processing pressure value of each processing device based on the status parameters, includes:

[0025] Based on the device hardware parameters, performance parameters characterizing the processing performance of each processing device are obtained;

[0026] Based on the equipment operating parameters, load parameters are obtained to characterize the load state of each processing device.

[0027] Based on the performance parameters and the load parameters, the processing pressure value of each processing device is obtained.

[0028] Furthermore, the device hardware parameters include the number of processors, processor frequency, disk speed, memory size, and network bandwidth; the performance parameters are obtained using the following formula:

[0029] Per j =r1×CPUn j ×(r2×Fre j +r3×DISKs j +r4×MEMs j +r5×NETw j )

[0030] Among them, Per j Let CPUn be the performance parameter of the j-th processing device in the cluster. j Let Fre be the number of processors in the j-th processing device in the cluster. j Let be the processor frequency of the j-th processing device in the cluster, and DISKs be the DISKs. j Let be the disk speed of the j-th processing device in the cluster, and MEMs j Let NETw be the memory size of the j-th processing device in the cluster. j R1 represents the network bandwidth of the j-th processing device in the cluster, and r1, r2, r3, r4, and r5 are all performance weighting coefficients.

[0031] Furthermore, the device operating parameters include processor utilization, memory utilization, disk utilization, and network throughput; the load parameters are obtained using the following formula:

[0032] Load j =t1×CPUu j +t2×MEMu j +t3×DISKu j +t4×NETt j

[0033] Among them, Load j Let CPUu be the load parameter of the j-th processing device in the cluster. j MEMU represents the processor utilization rate of the j-th processing device in the cluster. j Let DISKu be the memory utilization rate of the j-th processing device in the cluster.j NETt represents the disk utilization of the j-th processing device in the cluster. j t1 represents the network throughput of the j-th processing device in the cluster, and t1, t2, t3, and t4 are all load weighting coefficients.

[0034] Furthermore, the processing pressure value is obtained by the following formula:

[0035]

[0036] The beneficial effects of this invention are:

[0037] This invention provides a real-time big data processing method for massive data streams. It first acquires the status parameters of each processing device in the cluster, and then obtains the processing pressure value of each device based on these parameters. Next, it acquires the data stream to be processed within a preset time period. Then, based on the type and order of the data to be processed in the data stream, it establishes a processing priority for each piece of data. Finally, it allocates the data to be processed in the data stream to multiple processing devices for processing with optimal real-time performance. The real-time performance is determined based on the processing pressure value and the processing priority. Compared to existing technologies, this invention quantifies the current processing capacity of each device in the cluster as a processing pressure value, and then combines this with the priority and order of the data to be processed in the data stream to establish processing priorities. This allows each piece of data to be allocated to a suitable processing device based on the processing pressure value and processing priority, ensuring the real-time performance of data processing. Attached Figure Description

[0038] Figure 1 A flowchart illustrating an embodiment of the real-time big data processing method for massive data streams provided by the present invention.

[0039] Figure 2 for Figure 1 A flowchart of a method according to an embodiment of step S103. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0041] In computer science, a cluster is a group of interconnected computing resources (processing devices in this paper) that work together to complete a task or provide a service. These computers can be physical computers or virtual machines, connected together via a network and working collaboratively. Clusters are commonly used for processing large-scale data, high-performance computing, load balancing, and high availability. By distributing tasks across multiple computers in a cluster, higher computing performance and scalability can be achieved. Cluster technology is widely used in many fields, such as big data analytics, cloud computing, and web server clusters. By using clusters, higher computing efficiency, better performance, and stronger availability can be achieved. This invention primarily applies to load balancing clusters, which are used to distribute the load on a network or service. It improves overall performance and reliability by evenly distributing requests across the computers in the cluster.

[0042] Combination Figure 1 As shown in the figure, a specific embodiment of the present invention discloses a method for real-time big data processing of massive data streams, comprising:

[0043] S101. Obtain the status parameters of each processing device in the cluster, and obtain the processing pressure value of each processing device based on the status parameters;

[0044] S102. Obtain the data stream to be processed within a preset time period;

[0045] S103. Based on the type and order of the data to be processed in the data stream to be processed, establish a processing priority for each data to be processed.

[0046] S104. Distribute the data to be processed in the data stream to multiple processing devices for processing with optimal real-time performance, wherein the real-time performance is obtained based on the processing pressure value and processing priority.

[0047] Compared to existing technologies, this invention quantifies the current processing capacity of each device in the cluster into a processing pressure value, and then establishes a processing priority by combining the priority and order of the data to be processed in the data stream. In this way, each piece of data can be allocated to a suitable processing device for processing according to the processing pressure value and processing priority, thus ensuring the real-time performance of data processing.

[0048] Furthermore, in a preferred embodiment, the status parameters include device hardware parameters and device operating condition parameters. The device hardware parameters represent the device's hardware performance, while the device operating condition parameters indicate the device's current operating status. These two dimensions reflect the device's current data processing capability. It is understood that in practice, the specific parameters can be flexibly set according to the specific circumstances.

[0049] In this embodiment, step S101, obtaining the status parameters of each processing device in the cluster and obtaining the processing pressure value of each processing device based on the status parameters, specifically includes:

[0050] Based on the device hardware parameters, performance parameters characterizing the processing performance of each processing device are obtained;

[0051] Based on the equipment operating parameters, load parameters are obtained to characterize the load state of each processing device.

[0052] Based on the performance parameters and the load parameters, the processing pressure value of each processing device is obtained.

[0053] Specifically, in a preferred embodiment, the device hardware parameters include the number of processors, processor frequency, disk speed, memory size, and network bandwidth; the performance parameters are obtained by the following formula:

[0054] Per j =r1×CPUn j ×(r2×Fre j +r3×DISKs j +r4×MEMs j +r5×NETw j )

[0055] Among them, Per j Let CPUn be the performance parameter of the j-th processing device in the cluster. j Let Fre be the number of processors in the j-th processing device in the cluster. j Let be the processor frequency of the j-th processing device in the cluster, and DISKs be the DISKs. j Let be the disk speed of the j-th processing device in the cluster, and MEMs j Let NETw be the memory size of the j-th processing device in the cluster. j R1 represents the network bandwidth of the j-th processing device in the cluster, and r1, r2, r3, r4, and r5 are all performance weighting coefficients.

[0056] It is understandable that the number of processors, processor frequency, disk speed, memory size and network bandwidth mentioned above can all be represented by manually processed values ​​(such as normalization). The specific processing method can be flexibly set according to the actual situation, and will not be explained in detail in this article.

[0057] Specifically, in a preferred embodiment, the device operating parameters include processor utilization, memory utilization, disk utilization, and network throughput; the load parameters are obtained by the following formula:

[0058] Load j=t1×CPUu j +t2×MEMu j +t3×DISKu j +t4×NETt j

[0059] Among them, Load j Let CPUu be the load parameter of the j-th processing device in the cluster. j MEMU represents the processor utilization rate of the j-th processing device in the cluster. j Let DISKu be the memory utilization rate of the j-th processing device in the cluster. j NETt represents the disk utilization of the j-th processing device in the cluster. j t1 represents the network throughput of the j-th processing device in the cluster, and t1, t2, t3, and t4 are all load weighting coefficients.

[0060] Similarly, the processor utilization, memory utilization, disk utilization, and network throughput mentioned above can also be represented by manually processed values ​​(such as normalization). The specific processing method can be flexibly set according to the actual situation, and will not be explained in detail in this article.

[0061] Specifically, in a preferred embodiment, the processing pressure value is obtained by the following formula:

[0062]

[0063] Among them, S j Let be the processing pressure value of the j-th processing device in the cluster. The larger the value, the greater the processing pressure of the processing device. In this case, assigning high-priority data to it for processing is not a good choice.

[0064] Combination Figure 2 As shown, specifically, in a preferred embodiment, step S103, establishing a processing priority for each piece of data to be processed according to the type and order of the data to be processed in the data stream, specifically includes:

[0065] S201. Based on the message protocol type of the data to be processed, the type of data to be processed is obtained, wherein the type of data to be processed includes reliable data and unreliable data;

[0066] S202. Assign an initial priority to each type of data to be processed according to the type of data to be processed;

[0067] S203. Based on the sequential position of each data to be processed in the data stream to be processed, optimize the initial priority of each data to be processed to obtain the processing priority of each data to be processed.

[0068] In step S201 above, the type of data to be processed is determined based on the message protocol type. This type includes reliable data and unreliable data. Reliable data refers to data transmitted via a reliable protocol, such as TCP, while unreliable data is transmitted via an unreliable protocol, such as UDP. Reliable protocols require establishing a connection before transmission. Therefore, if reliable data is not processed promptly, the sending end will not receive a timely response and will repeatedly send multiple request messages. This undoubtedly increases the pressure on the network and cluster, affecting the real-time performance of other data processing. Therefore, reliable data needs to be processed first compared to unreliable data.

[0069] In step S202 above, according to the type of data to be processed, an initial priority is assigned to each type of data to be processed. The initial priority can be flexibly set according to specific needs. For example, the initial priority value of reliability data can be set to 0.6, and the initial priority value of reliability data can be set to 0.4. The larger the value, the higher the priority.

[0070] Furthermore, in step S203 above, the initial priority of each piece of data to be processed is optimized according to its sequential position in the data stream to be processed, and the processing priority of each piece of data to be processed is obtained according to the following formula:

[0071]

[0072] P i ∈[0,1]

[0073] P′ i ∈[0,1]

[0074] Among them, P i P′ represents the processing priority of the i-th data in the data stream to be processed. A larger value indicates a higher priority. i The initial priority of the i-th data to be processed in the data stream to be processed is given by k, where a larger value indicates a higher priority. k is the first adjustment parameter, i is the position of the data to be processed in the data stream to be processed, and n is the total number of data to be processed in the data stream to be processed.

[0075] The above process takes into account the position of the data to be processed in the data stream and optimizes the initial priority. k is an adjustment parameter used to control the degree to which the order of data in the data stream affects the priority. The larger k is, the more obvious the influence of the order of data in the data stream on the priority.

[0076] The above formula ensures that data with higher initial priority retains its higher priority, while also taking into account the order of data in the data stream. This guarantees that even data with higher initial priority at the end of the queue will receive an appropriate priority boost. This avoids the problem of delayed processing caused by high-priority data being at the end of the queue.

[0077] Furthermore, in a preferred embodiment, in step S104 above, the data to be processed in the data stream is distributed to multiple processing devices for processing with optimal real-time performance, wherein the real-time performance is obtained according to the following formula:

[0078]

[0079] S j ∈[0,1]

[0080] Among them, RT ij To ensure the real-time performance of allocating the i-th data item in the data stream to the j-th processing device in the cluster, a larger value of S indicates better real-time data processing. j α represents the processing pressure value of the j-th processing device in the cluster. The larger the value, the greater the processing pressure of the processing device. j is the number of the processing device in the cluster, and α is the second adjustment parameter.

[0081] In the above formula, α is used to adjust the influence of the processing pressure value and control the rate at which this factor's effect decays. The exponential function in the formula makes it more likely that higher-priority data will be assigned, while also allowing real-time calculations to consider the processing priority and processing pressure, as well as the continuous load of the equipment, providing a certain pressure suppression effect when the processing pressure is high.

[0082] Based on the real-time performance achieved through the above process, each piece of data to be processed can be assigned to a suitable processing device for processing, so as to take care of all the data on a macro level and maintain the real-time performance of processing massive data streams.

[0083] This invention provides a real-time big data processing method for massive data streams. It first acquires the status parameters of each processing device in the cluster, and then obtains the processing pressure value of each device based on these parameters. Next, it acquires the data stream to be processed within a preset time period. Then, based on the type and order of the data to be processed in the data stream, it establishes a processing priority for each piece of data. Finally, it allocates the data to be processed in the data stream to multiple processing devices for processing with optimal real-time performance. The real-time performance is determined based on the processing pressure value and the processing priority. Compared to existing technologies, this invention quantifies the current processing capacity of each device in the cluster as a processing pressure value, and then combines this with the priority and order of the data to be processed in the data stream to establish processing priorities. This allows each piece of data to be allocated to a suitable processing device based on the processing pressure value and processing priority, ensuring the real-time performance of data processing.

[0084] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for real-time processing of massive data streams, characterized in that, include: Obtain the status parameters of each processing device in the cluster, and obtain the processing pressure value of each processing device based on the status parameters; Acquire the data stream to be processed within a preset time period; Based on the type and order of the data to be processed in the data stream to be processed, a processing priority is established for each data to be processed. The data to be processed in the data stream is distributed to multiple processing devices for processing with optimal real-time performance, wherein the real-time performance is obtained based on the processing pressure value and the processing priority.

2. The real-time big data processing method for massive data streams according to claim 1, characterized in that, The step of establishing a processing priority for each piece of data to be processed based on the type and order of the data to be processed in the data stream includes: The type of data to be processed is determined based on the message protocol type of the data to be processed, wherein the types of data to be processed include reliable data and unreliable data; Based on the type of data to be processed, an initial priority is assigned to each type of data. Based on the sequential position of each piece of data to be processed in the data stream, the initial priority of each piece of data to be processed is optimized to obtain the processing priority of each piece of data to be processed.

3. The real-time big data processing method for massive data streams according to claim 2, characterized in that, The processing priority is obtained according to the following formula: P i ∈[0,1] P′ i ∈[0,1] wherein P i is the processing priority of the i-th to-be-processed data in the to-be-processed data stream, and a larger value represents a higher priority, P' i is the initial priority of the i-th to-be-processed data in the to-be-processed data stream, and a larger value represents a higher priority, k is a first adjustment parameter, i is the position of the to-be-processed data in the to-be-processed data stream, and n is the total number of to-be-processed data in the to-be-processed data stream.

4. The real-time big data processing method for massive data streams according to claim 3, characterized in that, The real-time performance is obtained according to the following formula: S j ∈[0,1] Among them, RT ij To ensure the real-time performance of allocating the i-th data item in the data stream to the j-th processing device in the cluster, a larger value of S indicates better real-time data processing. j α represents the processing pressure value of the j-th processing device in the cluster. The larger the value, the greater the processing pressure of the processing device. j is the number of the processing device in the cluster, and α is the second adjustment parameter.

5. The real-time big data processing method for massive data streams according to claim 4, characterized in that, The status parameters include device hardware parameters and device operating condition parameters; obtaining the status parameters of each processing device in the cluster, and obtaining the processing pressure value of each processing device based on the status parameters, includes: Based on the device hardware parameters, performance parameters characterizing the processing performance of each processing device are obtained; Based on the equipment operating parameters, load parameters are obtained to characterize the load state of each processing device. Based on the performance parameters and the load parameters, the processing pressure value of each processing device is obtained.

6. The real-time big data processing method for massive data streams according to claim 5, characterized in that, The device hardware parameters include the number of processors, processor frequency, disk speed, memory size, and network bandwidth; the performance parameters are obtained using the following formula: Per j =r1×CPUn j ×(r2×Fre j +r3×DISKs j +r4×MEMs j +r5×NETw j ) Among them, Per j Let CPUn be the performance parameter of the j-th processing device in the cluster. j Let Fre be the number of processors in the j-th processing device in the cluster. j Let be the processor frequency of the j-th processing device in the cluster, and DISKs be the DISKs. j Let be the disk speed of the j-th processing device in the cluster, and MEMs j Let NETw be the memory size of the j-th processing device in the cluster. j R1 represents the network bandwidth of the j-th processing device in the cluster, and r1, r2, r3, r4, and r5 are all performance weighting coefficients.

7. The real-time big data processing method for massive data streams according to claim 6, characterized in that, The device operating parameters include processor utilization, memory utilization, disk utilization, and network throughput; the load parameters are obtained using the following formula: Load j =t1×CPUu j +t2×MEMu j +t3×DISKu j +t4×NETt j Among them, Load j Let CPUu be the load parameter of the j-th processing device in the cluster. j MEMU represents the processor utilization rate of the j-th processing device in the cluster. j Let DISKu be the memory utilization rate of the j-th processing device in the cluster. j NETt represents the disk utilization of the j-th processing device in the cluster. j t1 represents the network throughput of the j-th processing device in the cluster, and t1, t2, t3, and t4 are all load weighting coefficients.

8. The real-time big data processing method for massive data streams according to claim 7, characterized in that, The processing pressure value is obtained by the following formula:

Images