Data processing method and system

Abstract

The invention discloses a data processing method and system, during real-time collection of data, non-functional requirements, such as high throughput, stability, easy expansion and high reliability, of the system are ensured through a big data cluster scheme, an infinite data stream is cut into a limited data stream according to quantity through setting of a counting window, efficient storage and calculation are realized, and the data processing efficiency is improved. According to the method, the calculation operation can be triggered when the data in the data stream reaches a certain amount, the real-time calculation service does not need to be restarted, and the time and resource overhead for storing the check point when the job stops and recovering the task from the check point when the job is restarted are saved. For the problem of data skew, in order to avoid overlarge difference of task processing speeds of different nodes and memory overflow, memory resources are reasonably allocated, and a random prefix method and a sampling skew Key method are provided to guarantee smooth execution of tasks. According to the method, the real-time data processing capacity is effectively improved, the concurrent processing design is optimized, and quick response under the high-concurrency condition is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing and relates to a data processing method and system. Background Art

[0002] In the process from project planning to completion, there are many key links involved, including project establishment, element guarantee, construction commencement, regulation and statistics, completion acceptance, commissioning, performance evaluation, etc. These links are intertwined, making project management a highly complex and systematic work. At present, governments at all levels and relevant functional departments generally face the following four major problems in project management: 1. The departments are not closely connected and the division of responsibilities is not clear: There is a lack of effective coordination among the functional departments, resulting in unclear responsibilities. The project investment and construction management mechanism urgently needs to be improved.

[0003] 2. The phenomenon of data silos is serious: There is insufficient data sharing among functional departments, the basic data is unclear, the caliber is inconsistent, and a unified project investment and management data system has not been established.

[0004] 3. Insufficient depth in project planning and lack of strategic projects: The existing mechanism is difficult to effectively identify and screen high-quality projects, resulting in an insufficient supply of high-quality and strategically significant projects.

[0005] 4. Outdated management methods and low efficiency: Project management at the city and county levels still mainly relies on traditional spreadsheets, lacks a unified digital management platform, and is difficult to conduct efficient data analysis and processing.

[0006] As projects expand in size and complexity, real-time uploading, processing and analysis of massive amounts of data have become key factors in project success. Traditional data management methods are no longer able to cope with the challenges of large amounts of data, diverse data types and high real-time requirements in current project management. In complex projects involving large-scale infrastructure construction, cross-departmental collaboration and multiple stakeholders, the timeliness, accuracy and traceability of data directly affect the progress and quality of the project.

[0007] There are many types of data involved in the project management process, including feasibility studies, resource allocation, market and policy analysis, capital and human resource management, construction progress and quality monitoring, as well as completion acceptance and performance evaluation. These data need to be efficiently transmitted and processed between departments and multiple links to ensure the orderly progress of project management. However, the current project management system faces the following significant problems when processing massive data and coping with high concurrent demands: 1) Limited data processing capabilities: Traditional databases and processing architectures are unable to quickly process large-scale project data, especially in real-time uploading and synchronization, which is prone to delays and bottlenecks, affecting the overall project progress.

[0008] 2) Insufficient system scalability: As the number of projects and the amount of data increase, the existing system is difficult to flexibly expand to meet new needs, causing the system to be overloaded and affecting stability and reliability.

[0009] 3) Lack of high-concurrency processing capabilities: At critical nodes of the project or in data-intensive operations, the existing system cannot effectively respond to a large number of requests and is prone to crashes and data loss risks.

[0010] 4) Insufficient data integration and analysis capabilities: Data from different sources use different formats and standards. The existing system has bottlenecks in integration and analysis, making it difficult to quickly generate valuable decision-making reference results. Summary of the invention

[0011] The purpose of the present invention is to solve the problem that the real-time data processing capability and concurrent processing capability of the project management system in the prior art are insufficient, resulting in low overall system performance and reliability, and difficulty in coping with the complexity and diversity challenges existing in current project management, and to provide a data processing method and system.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions: A data processing method comprises the following steps: Collect project data in real time and write the collected project data into the message queue Kafka; Set the number of counting windows. By setting the windows, the infinite project data flow is cut into a finite data flow. When the collected data reaches the window, the data is saved. When the number of windows reaches the set number, the data in the window is connected and calculated. Based on the connection calculation results, the data is grouped and aggregated, and the data skew problem is processed during the aggregation calculation to complete the data classification processing.

[0013] A further improvement of the present invention is: The real-time collection of project data and writing the collected project data into the message queue Kafka includes the following steps: Complete project data collection through Flume, including: Start the data collection task on the remote project data server to correspond to the number of nodes where the Flume data collection task is deployed in the local cluster; Start the data collection task on each node in the local cluster and send the collected data to the distributed message queue Kafka.

[0014] The project data collection by Flume also includes the following steps: Among the nodes of the local cluster, one node is selected as a disaster recovery node, and the disaster recovery node subscribes to and collects all project data file directories.

[0015] The window also includes the following steps when receiving data: During off-peak or peak hours, combined with nacos dynamic configuration calculation tasks, the counting window size is adjusted in real time, that is, during off-peak hours, the counting window is reduced.

[0016] The method of saving the collected data after it reaches the window includes the following steps: When the collected project data reaches the counting window, a real-time project data stream and a data stream in the business table are formed; By opening two State areas for data storage in the stream processing framework Flink, the real-time project data stream is stored in LState, and the data stream in the business table is stored in RState; When performing a connection calculation, the project data flow will be connected to the data flow in the business table for calculation, or the data flow in the business table will be connected to the project data flow for calculation, and the calculation results will be sent to the downstream operator to complete the data reading.

[0017] The processing of the data skew problem includes: When the amount of tilted data is small, adjust the memory size occupied by a single task; When the amount of skewed data is large, data skew is handled by random prefix method and sampling skew method.

[0018] The method of processing the data skew problem by using the random prefix method and the sampling skew method comprises the following steps: The random prefix method first locates the location where the skew occurs, that is, locates the corresponding operator in the process of generating data skew. By adding a random prefix before the skewed key, the data of the skewed project is scattered to different tasks for execution, and then the aggregation operation is performed within one task. The aggregation operation includes: First, perform local aggregation and add a random prefix to the key of each skewed data. That is, the skewed key will become multiple keys. Perform shuffle operation on the data after adding random prefix to complete local aggregation; Before the second aggregation, the random prefix is ​​removed, and then global aggregation is performed again to obtain the final result; The sampling skew method handles the data skew problem by sampling skew keys, including the following steps: The sampling tilt key method is applied to the data tilt problem caused by connection calculation; The massive data corresponding to the key is split into multiple RDDs, and the data is scattered by adding random prefixes. At this time, the data corresponding to the tilted key will be distributed to multiple task nodes for connection calculation.

[0019] A data processing system, comprising: Collect project data in real time and write the collected project data into the message queue Kafka; Set the number of counting windows. By setting the windows, the project data is truncated into a limited data stream. When the collected data reaches the window, the data is saved. When the number of data in the window reaches the set number, the data in the window is connected and calculated. Based on the connection calculation results, the data is grouped and aggregated, and the data skew problem is processed during the aggregation calculation to complete the data classification processing.

[0020] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0021] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any method described in the present invention.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a data processing method. When data is collected in real time, a big data cluster solution is used to ensure the non-functional requirements of the system, such as high throughput, stability, easy expansion, and high reliability. By setting a counting window, an infinite data stream is truncated into a finite data stream according to quantity, so as to achieve efficient storage and calculation. When the data in the data stream reaches a certain amount, a calculation operation can be triggered without restarting the real-time calculation service, and the time and resource overhead of saving checkpoints when the job is stopped and restoring tasks from the checkpoints when the job is restarted is eliminated. When the data is aggregated and calculated, the data skew problem is handled to avoid memory overflow. The smooth execution of tasks is guaranteed by reasonably allocating memory resources. The method improves the real-time data processing capability, optimizes the concurrent processing design, ensures a fast response under high concurrency, and significantly enhances the scalability of the system. It can flexibly adapt to the scale and complexity of different projects, provide more accurate and timely project data analysis, and support efficient decision-making.

[0023] Furthermore, in the present invention, among the nodes of the local cluster, one node is selected as a disaster recovery node. When a data collection task on other nodes fails, the "safe node" can be used to continue the collection of all data and the backtracking of data during the downtime of the remaining nodes. When the number of nodes in the cluster is sufficient, the same data collection task can also be deployed on the remaining nodes, and multiple machines can collect all real-time project data to achieve higher reliability. In this way, a higher data throughput can be achieved, and the disaster recovery backup strategy can ensure the high reliability of data and the stability of services in the event of a small-scale downtime in the cluster.

[0024] Furthermore, in the present invention, when the amount of tilted data is small, by adjusting the memory size of a single task, it is ensured that when data tilt occurs, memory overflow is not likely to occur, and sufficient memory resources are allocated to ensure the completion of task execution. Excessive allocation of memory will lead to the waste of some resources, and resources are exchanged for the high reliability of the system. Therefore, when the amount of tilted data is too large, in order to avoid the waste of resources, the present invention proposes a random prefix method and a sampling tilt method for processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 It is the overall structure diagram of the model of the present invention; Figure 2 This is a cluster service architecture diagram of the real-time project data collection part of the present invention; Figure 3 This is a diagram of the cluster service architecture for the real-time project computing part of the present invention; Figure 4 It is a schematic diagram of dual-stream connection data storage of the present invention; Figure 5 It is a schematic diagram of the dual-flow connection of the present invention; Figure 6 Tilt instances for data; Figure 7 Schematic diagram of the data tilt-random prefix method of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0030] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0031] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0032] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings: See also Figure 1, this embodiment discloses a data processing method, specifically a method for improving the real-time data processing capability and concurrent processing capability of the system based on big data technology, which includes three parts: reliable big data cluster design, real-time data dual-stream connection design and data tilt solution. For real-time data collection tasks, by starting multiple data collection tasks on a remote data server, corresponding to the number of nodes of the real-time log collection framework Flume data collection task deployed in the local cluster, different real-time project data file directories are subscribed and collected respectively. One of the nodes is selected as a disaster recovery node in the data collection stage to subscribe to all real-time project data file directories for the collection of all data, and disaster recovery backup strategies such as data backtracking are performed when other nodes are down. And the collected data is put into the message queue Kafka, which effectively ensures the high reliability of data and the stability of services. For the collected real-time data, the real-time data is obtained from the message queue, and the infinite data stream is truncated into a finite data stream by setting a counting window. When the data volume reaches the specified threshold, the calculation operation is triggered. And the size of the counting window can be dynamically adjusted according to the peak and valley periods to optimize resource utilization and reduce the time and resource consumption when the job is restarted and recovered. After the data reaches the counting window, the system opens two state storage branches to store real-time project flow data and business flow data respectively. The system performs connection calculations on it and sends the calculation results to the downstream to ensure real-time and stability when processing infinite data streams. For data skew phenomena caused by calculation operations such as grouping and aggregation in the business, in order to avoid memory overflow, the system adjusts the memory size of a single task to ensure that sufficient memory resources are allocated when the data is skewed to support the smooth execution of the task. In addition, the random prefix method or the sampling tilt key method can be used to disperse the data into multiple tasks for connection calculations, and finally aggregate the output results, which can effectively solve the problems of insufficient real-time data processing capabilities and concurrent processing capabilities of existing project management systems. The overall performance and reliability are low, and it is difficult to cope with the complexity and diversity challenges in current project management.

[0034] Embodiment 1, a data processing method, comprising the following steps: Step 1: Use the real-time log collection framework Flume to complete the collection of massive project data through the big data cluster. Start multiple data collection tasks on the remote server, subscribe to different real-time project data file directories, and write the collected data to the message queue Kafka. Select a node as a disaster recovery node, subscribe to and collect all project data file directories, and backtrack the data when the remaining nodes go down, to achieve a high-throughput, stable, easy-to-expand, and highly reliable big data cluster.

[0035] For details, see Figure 2 , the specific steps are: First, start three data collection tasks on the remote project data server, corresponding to the number of nodes where Flume data collection tasks are deployed in the local cluster, and subscribe to different real-time project data file directories respectively; Then, the data collection task is started on each node in the local cluster, and different nodes collect data in different file directories.

[0036] This step also includes the setting of "Security Node": In the cluster, the Hadoop003 node with fewer services deployed, that is, with lower load, is set as a "safe node" and a separate data collection task is started to subscribe to the real-time project data files in all file directories. Since this node has fewer services deployed and sufficient resources such as memory and hard disk, it can be used as a disaster recovery node in the data collection stage. When the data collection tasks on the Hadoop001 and Hadoop002 nodes fail, the Hadoop003 node can continue to collect all data and trace back the data of the other nodes during the downtime stage. Similarly, when the number of nodes in the cluster is sufficient, the same data collection task can be deployed on the other nodes, and multiple machines can collect all real-time project data to achieve higher reliability.

[0037] In this step, the resources allocated to a single data collection task are fixed. The collection tasks that subscribe to different file directories on Hadoop001, Hadoop002, and Hadoop003 have higher execution efficiency. The real-time collected project data can be pushed to the distributed message queue through KafkaSink. The data collection task resources that subscribe to all real-time project data file directories on Hadoop003 are the same as the data collection task resources that subscribe to different directories. However, since its main function is disaster recovery, there is no real-time requirement.

[0038] Based on the above design, by having multiple machines Hadoop001, Hadoop002, and Hadoop003 simultaneously subscribe to different file directories in the remote project data server, distributed collection of real-time project data can be achieved to achieve a higher data throughput, and the disaster recovery backup strategy can ensure high data reliability and service stability in the event of a small-scale downtime in the cluster.

[0039] See also Figure 3 The real-time project data collected by each node will be pushed into the distributed message queue through KafkaSink.

[0040] Specifically, when designing a distributed message queue to store real-time project data, the embodiment of the present invention also divides the number of partitions and partition copies under the topic according to the number of nodes in the cluster. In Kafka, data is divided by topic, and multiple partitions correspond to one topic. Taking the above-mentioned cluster of three nodes as an example, one topic corresponds to three partitions, and each partition has three copies, and its copy hashes are stored on different nodes. Each partition elects a leader to provide services, namely, the writing service of real-time project data and the data pulling service of real-time project computing tasks. In the case of data writing to the partition, after the leader partition is written successfully, that is, the writing is successful, the leader will synchronize the written data to its backup partition through the network for disaster recovery backup. Similarly, when the real-time project computing task consumes real-time project data, it also reads the data in the leader partition. When the node where the leader partition is located is down, the cluster used by this system uses Zookeeper to implement the election of a new leader partition, that is, a leader is elected from the other two surviving partitions to continue to provide services to the outside world. At the same time, Kafka itself saves the offset of the consumed project data to ensure that the project data will not be consumed and calculated repeatedly. The real-time project calculation function will submit the calculation task to the cluster, and multiple nodes will start the calculation task. The calculation task on each node will subscribe to the leader partition under the corresponding topic in Kafka. The calculation task will pull the data of the leader partition in Kafka in real time for consumption calculation. After the data in each window is calculated and successfully written to the memory cache, the consumption offset will be submitted to Kafka. The next consumption calculation will pull new data from the next position of the last submitted offset. The real-time calculation task will take a snapshot of the calculation state at the current time point at regular intervals, that is, the save point, and persist the current calculation state locally. After a node in the cluster crashes and restarts, the calculation task can be restored from the most recent save point state to avoid consumption from the offset of 0. According to the above design, distributed message queues are deployed on multiple machines, and the real-time project data collected by KafkaSink on different nodes are written to different partitions under the same topic of Kafka, and the data writing operation is dispersed to different nodes. Flink computing tasks are deployed on different nodes, and real-time project data from different partitions under the same topic are pulled for consumption computing, which can achieve a higher data throughput. The partition disaster recovery backup strategy and leader partition service strategy in the distributed message queue, as well as the save point of the real-time project computing task, can also ensure high data reliability and service stability when some nodes in the cluster go down.

[0041] Step 2: When pulling real-time project data from the distributed message queue Kafka for consumption, since the real-time project data stream and the data stream in the business table are infinite, the infinite data stream is truncated into a finite data stream by volume through dynamic window settings to achieve efficient storage and calculation.

[0042] The real-time project process data in the distributed message queue will be continuously pulled for consumption and connected with the data stream in the business table for real-time calculation. In order to ensure the real-time performance of the system, timely output data, and avoid errors caused by inconsistent generation speeds of the two data streams, the present invention uses a counting window to truncate the infinite data stream into a finite data stream according to the amount, and trigger the calculation operation when the real-time data in the counting window reaches a certain amount.

[0043] Specifically, during off-peak or peak periods, by combining with nacos, computing jobs can be dynamically configured and the size of the counting window can be set. That is, during off-peak periods, the counting window can be adjusted to a smaller size without having to wait too long to satisfy the larger counting window set previously. This design does not require restarting the real-time computing service, eliminating the time and resource overhead of saving checkpoints when the job stops and restoring tasks from the saved checkpoints when the job restarts.

[0044] In this step, the proposed dual-stream connection has the following advantages compared with the traditional database table connection: (1) The data sets on both sides of the join operation are infinite data. In traditional database connections, the data sets of the two tables on the left and right are limited. However, in the dual-stream connection scenario, the data on both sides will flow in continuously.

[0045] (2) The results generated by the connection calculation are continuously generated and updated. Traditional database table connections are executed once to produce the final result and then exit. Dual-stream connections will continuously generate new results.

[0046] (3) Query calculation is driven by both sides. Due to the inconsistent data generation speed of the left and right sides of the dual-stream connection, the right side data may not arrive when the left side data arrives, or the left side data may not arrive when the right side data arrives. Therefore, the left and right side stream data are saved during implementation to ensure the semantics of the connection.

[0047] Step 3: After determining the counting window, when the real-time project data and the data in the business table reach the window, use the dual-stream connection design to save the data and perform corresponding calculations on it, and send the calculation results to the downstream operator to ensure seamless data transmission.

[0048] For detailed calculation results, see Figure 4After the counting window is determined, when the data arrives in the window, it is necessary to save the data. That is, save the data arriving in the counting window on the two data streams. Open two States (state areas) in Flink (stream processing framework) for data storage. After the data of the left and right streams arrive, perform the following operations: After the LeftEvent (left event, i.e. real-time project flow data) arrives, it is stored in LState (left state area), and after the RightEvent (right event, i.e. data in the business table) arrives, it is stored in RState (right state area). Then LeftEvent will go to RState for connection calculation and send the result after calculation to the downstream operator, and RightEvent will go to LState for connection calculation and also send the result after calculation to the downstream operator.

[0049] See also Figure 5 , the real-time project data stream and the data stream in the business table continuously generate data. The numbers on the dotted line are the order in which the data enters the window. The data is stored in LState and RState. When the real-time data streams (plate1, content1) and (plate1, content2) on the left enter, the data (plate1, content6) in the business table on the right has not yet entered. At this time, there is no data on the right side that can be connected and calculated. The real-time project data stream on the left will be sent to the downstream and the business stream data on the left will be filled with NULL. When the same key plate1 appears in the right data stream, the event with NULL that arrives first on the left will be withdrawn. Corresponding to the +, - in the figure, + represents the forward record, and - represents the withdrawn record. When the right data flows in and finds that there is already a key that can be connected on the left, and it is the first right data that can be connected, it is necessary to withdraw the NULL record sent from the left, and send the complete data after the connection calculation to the downstream. The subsequent connection operations are also in the order of data arrival in the window.

[0050] Step 4: When performing group aggregation calculations on data, for data skew problems, the system can adjust the memory size for a single task to ensure that memory overflow is avoided when data skew occurs, and ensure the smooth execution of tasks by reasonably allocating memory resources.

[0051] See also Figure 6 , Figure 6This is a simple example of data tilt. The project corresponding to the national AXXXXX project number has a total of 7 data in all tasks. All data of this project will be distributed to a single downstream task for calculation, while the provincial AXXXXX and municipal AXXXXX projects each have one data in all tasks, so each downstream task only needs to process one data. The amount of data in the first downstream task is seven times the amount of data in the remaining tasks, and its calculation time is longer than that of the remaining tasks, and the execution speed of a stage is determined by the slowest task.

[0052] During the execution of a job, even if there are sufficient computing resources in the cluster, when facing a large amount of data, huge overhead will still be incurred when performing shuffle (data distribution across nodes and processes within the cluster), because it includes disk I / O, data serialization, network I / O overhead, and organizing data for shuffle (data distribution across nodes and processes within the cluster).

[0053] The simplest method is to expand resources and adjust the memory size of a single task. This method ensures that memory overflow is not likely to occur when data skew occurs, and guarantees task execution completion by allocating sufficient memory resources. This method is suitable when the amount of skewed data is small. Excessive allocation of memory will lead to waste of some resources, and resources are exchanged for high system reliability.

[0054] Step 5: For the data skew problem, you can also use the random prefix method and sampling skew key to split the connection operation, and disperse the massive data corresponding to the key into multiple RDDs (Resilient Distributed Datasets) to prevent a few tasks from causing memory overflow due to processing a large amount of data, thereby improving the overall task execution efficiency and ensuring the high reliability of the system.

[0055] For details, see Figure 7 The present invention proposes a random prefix method, which needs to locate the location where the skew occurs first, that is, locate the corresponding operator in the stage where the data skew occurs. Before generating the skewed key, the data of the skewed project is scattered to different tasks for execution by adding a random prefix, and finally the aggregation operation is performed within one task.

[0056] The implementation idea of ​​this method is to perform aggregation in two stages. First, perform local aggregation and add a random prefix, such as a random number, to the key of each data that has skewed. At this time, the skewed key will become multiple keys, such as Figure 5In China AXXXXX, 1) (Country AXXXXX, 1) (Country AXXXXX, 1) (Country AXXXXX, 1), it will become (1_Country AXXXXX, 1) (1_Country AXXXXX, 1) (2_Country AXXXXX, 1) (2_Country AXXXXX, 1). When performing a shuffle operation on the data with a randomly added prefix, it will first perform local aggregation, and the result is (1_Country AXXXXX, 2) (2_Country AXXXXX, 2). Before the second aggregation, the randomly added prefix is removed, and the result is (Country AXXXXX, 2)(Country AXXXXX, 2). Finally, a global aggregation operation is performed again to obtain the result (Country AXXXXX, 4).

[0057] Regarding the data skew caused by join calculations, the present invention proposes sampling the skewed keys and splitting the join operation. For this type of data skew, it is also the case that a small number of item numbers correspond to an overly large amount of data. The massive data corresponding to these keys is split into multiple RDDs (Resilient Distributed Datasets), and the data is shuffled by adding a random prefix in the same way. At this time, the data corresponding to the skewed keys will be distributed to multiple tasks for join calculations. This method expands the data of a small number of skewed keys without expanding the data corresponding to all keys, saving memory resources.

[0058] Regarding the method of sampling skewed keys, this method is applicable to the data skew caused by join calculations.

[0059] The method of sampling skewed keys samples the data in the skewed RDD (Resilient Distributed Dataset) that is generated to obtain a sample data. The data volume corresponding to different keys in the sample is counted to obtain the keys that cause the skew. The data corresponding to these keys is separately generated into an RDD (Resilient Distributed Dataset), and a random prefix is added before this data. The other keys that do not cause skew are generated into another RDD (Resilient Distributed Dataset), and the data corresponding to the skewed keys statistically obtained in the sampling stage in this RDD (Resilient Distributed Dataset) is filtered out to generate a new RDD (Resilient Distributed Dataset). By adding a random prefix within n before each piece of data, the data volume of a single key is expanded by n times. The data in the RDD (Resilient Distributed Dataset) with a randomly added prefix before is joined with the data in this RDD (Resilient Distributed Dataset). At this time, the data of the originally same key is scattered into n parts and distributed to multiple tasks for join calculations. The other two ordinary RDDs (Resilient Distributed Datasets) perform ordinary join calculations. Finally, the results of the two joins are merged once to obtain the final join result, such as Figure 6The following is an example of this method. Sample the project data in the skewed RDD (Resilient Distributed Dataset) to obtain the project numbers corresponding to the skewed data, Country AXXXXX, Country BXXXXX, Country CXXXXX, add a random prefix before the project number, expand the data of the corresponding project number in another RDD to be connected, and also add a random prefix, and then perform the connection calculation to solve the data skew caused by the connection calculation.

[0060] With the help of big data technology, this method not only improves the real-time data processing capability of the system, but also optimizes the concurrent processing design to ensure rapid response under high concurrency. In addition, the scalability of the system is significantly enhanced, and it can flexibly adapt to the scale and complexity of different projects. Based on the analysis and prediction model of big data, the system can provide more accurate and timely project data analysis and support efficient decision-making.

[0061] Example 2 The embodiment of the present invention further discloses a data processing system, including: Collect project data in real time and write the collected project data into the message queue Kafka; Set the number of counting windows. By setting the windows, the project data is truncated into a limited data stream. When the collected data reaches the window, the data is saved. When the number of windows reaches the set number, the data in the window is connected and calculated. Based on the connection calculation results, the data is grouped and aggregated, and the data skew problem is processed during the aggregation calculation to complete the data classification processing.

[0062] A schematic diagram of a terminal device provided in an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0063] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0064] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0065] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0066] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0067] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0068] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The following steps are involved: Collect project data in real time and write the collected project data into the message queue Kafka; Set the number of counting windows. By setting the windows, the infinite project data flow is cut into a finite data flow. When the collected data reaches the window, the data is saved. When the number of windows reaches the set number, the data in the window is connected and calculated. Based on the connection calculation results, the data is grouped and aggregated, and the data skew problem is processed during the aggregation calculation to complete the data classification processing.

2. A data processing method according to claim 1, characterized in that: The real-time collection of project data and writing the collected project data into the message queue Kafka includes the following steps: Complete project data collection through Flume, including: Start the data collection task on the remote project data server to correspond to the number of nodes where the Flume data collection task is deployed in the local cluster; Start the data collection task on each node in the local cluster and send the collected data to the distributed message queue Kafka.

3. A data processing method according to claim 2, characterized in that: The project data collection by Flume also includes the following steps: Among the nodes of the local cluster, one node is selected as a disaster recovery node, and the disaster recovery node subscribes to and collects all project data file directories.

4. A data processing method according to claim 1, characterized in that: The window also includes the following steps when receiving data: During off-peak or peak hours, combined with nacos dynamic configuration calculation tasks, the counting window size is adjusted in real time, that is, during off-peak hours, the counting window is reduced.

5. A data processing method according to claim 1, characterized in that: The method of saving the collected data after it reaches the window includes the following steps: When the collected project data reaches the counting window, a real-time project data stream and a data stream in the business table are formed; By opening two State areas for data storage in the stream processing framework Flink, the real-time project data stream is stored in LState, and the data stream in the business table is stored in RState; When performing a join calculation, the project data stream will be joined with the data stream in the business table for calculation, or the data stream in the business table will be joined with the project data stream for calculation, and the calculation results will be sent to the downstream operator to complete the data reading.

6. A data processing method according to claim 1, characterized in that: The processing of the data skew problem includes: When the amount of tilted data is small, adjust the memory size occupied by a single task; When the amount of skewed data is large, data skew is handled by random prefix method and sampling skew method.

7. A data processing method according to claim 6, characterized in that: The method of processing the data skew problem by using the random prefix method and the sampling skew method comprises the following steps: The random prefix method first locates the location where the skew occurs, that is, locates the corresponding operator in the process of generating data skew. By adding a random prefix before the skewed key, the data of the skewed project is scattered to different tasks for execution, and then the aggregation operation is performed within one task. The aggregation operation includes: First, perform local aggregation and add a random prefix to the key of each skewed data. That is, the skewed key will become multiple keys. Perform shuffle operation on the data after adding random prefix to complete local aggregation; Before the second aggregation, the random prefix is ​​removed, and then global aggregation is performed again to obtain the final result; The sampling skew method handles the data skew problem by sampling skew keys, including the following steps: The sampling tilt key method is applied to the data tilt problem caused by connection calculation; The massive data corresponding to the key is split into multiple RDDs, and the data is scattered by adding random prefixes. At this time, the data corresponding to the tilted key will be distributed to multiple task nodes for connection calculation.

8. A data processing system, characterized in that: include: Collect project data in real time and write the collected project data into the message queue Kafka; Set the number of counting windows. By setting the windows, the project data is truncated into a limited data stream. When the collected data reaches the window, the data is saved. When the number of data in the window reaches the set number, the data in the window is connected and calculated. Based on the connection calculation results, the data is grouped and aggregated, and the data skew problem is processed during the aggregation calculation to complete the data classification processing.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images