Data aggregation processing method, apparatus, device, and storage medium

Abstract

The application belongs to the technical field of data processing, and discloses a data aggregation processing method, device, equipment and storage medium; the application comprises the following steps: obtaining original service data of each upstream application, partitioning the original service data to obtain a plurality of partition data; distributing each partition data to each different one-layer node, processing the partition data through the one-layer node to obtain a plurality of reference service data; converging the plurality of reference service data to a two-layer node, aggregating the plurality of reference service data through the two-layer node to obtain aggregated data; and sending the aggregated data to a downstream application; the application provides the processing capacity of a large batch of service data through a plurality of one-layer nodes, can quickly expand the processing capacity when the upstream data volume changes, and further aggregates the data through the two-layer node to realize the forwarding of a large amount of original service data of the upstream, thereby avoiding the problems of data delay and loss caused by processing a large amount of data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data aggregation processing method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid popularization of cloud computing and the internet, enterprises are gradually undergoing intelligent transformation in various business operations, generating a large amount of business-related data. How to process this data in real-time, efficiently, and stably, and transform it into business-meaning metrics, has become a problem that enterprises need to solve. Normally, upstream applications send business data directly to downstream applications via Kafka. This method requires further aggregation and processing of the business data by the upstream application, which adds related processing logic and impacts its performance. Furthermore, whenever the downstream requires new data, the upstream application needs to be modified, inserting relevant data collection and processing points into the corresponding business logic. This requires development and testing personnel to spend time and effort on development and testing, and it cannot respond to downstream needs in real time, requiring a considerable wait before production can begin sending data.

[0003] If an intermediate application directly receives raw data from the upstream application, processes it, and then sends it to the downstream, the development of the upstream application can be greatly simplified without affecting its performance. By reserving relevant configuration functions in the intermediate application and configuring the processing logic in the intermediate application, downstream needs can be responded to quickly.

[0004] However, this approach presents several challenges. The upstream sends raw data directly through Kafka, resulting in a significantly larger volume of raw data compared to the final processed data—potentially by several orders of magnitude. This places immense demands on the data processing capabilities of the intermediate application. Poorly designed intermediate applications can lead to data delays and data loss. Furthermore, the added layer of application means that if a node in the intermediate application fails, the downstream system will not receive data, reducing its robustness. Therefore, ensuring the availability of the intermediate application is crucial. Summary of the Invention

[0005] The main objective of this invention is to provide a data aggregation processing method, apparatus, device, and storage medium, which aims to solve the technical problem that data delay and loss are easily generated when intermediate applications forward and process large amounts of data from upstream applications.

[0006] To achieve the above objectives, the present invention provides a data aggregation processing method, the method comprising the following steps:

[0007] Obtain the raw business data from each upstream application, partition the raw business data, and obtain multiple partition data.

[0008] The data of each partition is allocated to different first-level nodes, and the partition data is processed by the first-level nodes to obtain multiple reference business data.

[0009] The multiple reference business data are aggregated to a second-level node, and the multiple reference business data are aggregated through the second-level node to obtain aggregated data;

[0010] The aggregated data is sent to downstream applications.

[0011] Optionally, after allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the method further includes:

[0012] Query the topic data processing table to obtain the data being processed and the total amount of data that can be processed by the target processing node. The target processing node is any one of the nodes in the first layer.

[0013] The presence of spare data processing capacity of the target processing node is determined based on the amount of data being processed and the total amount of data that can be processed by the target processing node.

[0014] When the target processing node has spare data processing capacity, the topic pending data table is queried to obtain pending data that meets the preset requirements. The pending data is then allocated to the target processing node. The topic pending data table includes data information for each topic, including pending data, number of partitions, and total number of processing nodes.

[0015] Optionally, when the target processing node has spare data processing capacity, querying the topic pending data table, obtaining pending data that meets preset requirements from the topic pending data table, and allocating the pending data to the target processing node includes:

[0016] When the target processing node has spare data processing capacity, query the topic pending data table to determine whether the total number of partitions of the topic pending data in the topic pending data table is greater than the total number of processing nodes of the topic pending data.

[0017] If the number of partitions is greater than the total number of processing nodes, then query the topic data processing table to determine whether the target processing node processes the partition data of the topic of the data to be processed;

[0018] If the target processing node does not have partition data for the topic of the data to be processed, then the data to be processed will be assigned to the target processing node.

[0019] Optionally, after allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the method further includes:

[0020] Query the topic data processing table, and determine whether the topic of the current thread of the target node is a topic to be processed based on the query results;

[0021] If the topic of the current thread of the target node is a topic to be processed, then determine whether the topic processing record of the current thread of the target processing node exists. If it does not exist, stop processing the topic data to be processed of the current thread of the target processing node.

[0022] If it exists, update the node heartbeat of the target processing node and update the total number of partitions of the topic where the target processing node is located.

[0023] Optionally, after allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the method further includes:

[0024] Obtain the heartbeat record of the target processing node, and determine whether the heartbeat timeout of the target processing node is determined based on the heartbeat record;

[0025] If the heartbeat of the target processing node times out, the processing record of the unprocessed data of the target processing node is deleted;

[0026] Obtain the total number of partitions in the upstream application corresponding to the target processing node, and determine whether the total number of partitions is less than the total number of nodes in the first layer. If the total number of partitions is less than the total number of nodes in the first layer, randomly delete the data processing records of N target processing nodes.

[0027] Optionally, after aggregating the multiple reference service data to a Layer 2 node and obtaining aggregated data through the Layer 2 node, the method further includes:

[0028] Obtain the data being processed and the total amount of data that can be processed by the target aggregation node, wherein the target aggregation node is any one of the second-level nodes;

[0029] Based on the amount of data being processed and the total amount of data that the target aggregation node can process, determine whether the target aggregation node has spare data aggregation capabilities;

[0030] When the target processing node has spare data processing capacity, the topic data aggregation table is queried to obtain the data to be aggregated that meets the preset requirements. The data to be aggregated is then assigned to the target aggregation node. The topic data aggregation table includes data information for each topic, and the data information includes the data to be processed and the total number of processing nodes.

[0031] Optionally, after aggregating the multiple reference service data to a Layer 2 node and obtaining aggregated data through the Layer 2 node, the method further includes:

[0032] Query the data table of topics to be aggregated, and determine whether the topic of the current thread of the target aggregation node is a topic to be processed based on the query results;

[0033] If the topic of the current thread of the target node is a topic to be processed, then determine whether the topic processing record of the current thread of the target aggregation node exists. If it does not exist, stop processing the topic data to be processed of the current thread of the target processing node.

[0034] If it exists, update the node heartbeat of the target processing node, where the target aggregation node is any node in the second-level nodes.

[0035] Optionally, after aggregating the multiple reference service data to a Layer 2 node and obtaining aggregated data through the Layer 2 node, the method further includes:

[0036] Obtain the heartbeat record of the target aggregation node, and determine whether the heartbeat of the target aggregation node has timed out based on the heartbeat record;

[0037] If the heartbeat of the target aggregation node times out, the processing record of the pending data of the target aggregation node is deleted.

[0038] Furthermore, to achieve the above objectives, the present invention also proposes a data aggregation processing apparatus, the data aggregation processing apparatus comprising:

[0039] The data processing module is used to acquire raw business data from various upstream applications, partition the raw business data, and obtain multiple partition data.

[0040] The data processing module is also used to allocate the data of each partition to different first-level nodes, and process the partition data through the first-level nodes to obtain multiple reference business data.

[0041] The data aggregation module is used to aggregate the multiple reference business data to the second-level node, and then aggregate the multiple reference business data through the second-level node to obtain aggregated data;

[0042] The data aggregation module is also used to send the aggregated data to downstream applications.

[0043] Furthermore, to achieve the above objectives, the present invention also proposes a data aggregation processing device, which includes: a memory, a processor, and a data aggregation processing program stored in the memory and executable on the processor, wherein the data aggregation processing program is configured to implement the steps of the data aggregation processing method described above.

[0044] This invention provides the ability to process ultra-large batches of business data through multiple first-layer nodes. It can also dynamically configure the number of processing nodes required by the corresponding business based on the different business data volume. It can quickly expand the processing capacity when the upstream data volume changes. Then, the data is aggregated through second-layer nodes to realize the forwarding of a large amount of raw business data from upstream, avoiding the problems of data delay and loss that are easy to occur when processing a large amount of data. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the structure of the data aggregation and processing device in the hardware operating environment involved in the embodiments of the present invention;

[0046] Figure 2 This is a flowchart illustrating the first embodiment of the data aggregation processing method of the present invention;

[0047] Figure 3 This is a schematic diagram of the data aggregation processing system structure according to an embodiment of the data aggregation processing method of the present invention;

[0048] Figure 4 This is a flowchart illustrating the second embodiment of the data aggregation processing method of the present invention;

[0049] Figure 5 This is a flowchart illustrating the third embodiment of the data aggregation processing method of the present invention;

[0050] Figure 6 This is a structural block diagram of the first embodiment of the data aggregation and processing device of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the data aggregation processing device structure in the hardware operating environment involved in the embodiments of the present invention.

[0054] like Figure 1 As shown, the data aggregation and processing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the data aggregation processing device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data aggregation processing program.

[0057] exist Figure 1 In the data aggregation processing device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the data aggregation processing device of the present invention can be set in the data aggregation processing device, and the data aggregation processing device calls the data aggregation processing program stored in the memory 1005 through the processor 1001 and executes the data aggregation processing method provided in the embodiment of the present invention.

[0058] This invention provides a data aggregation processing method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of a data aggregation processing method according to the present invention.

[0059] In this embodiment, the data aggregation processing method includes the following steps:

[0060] Step S10: Obtain the raw business data of each upstream application, partition the raw business data, and obtain multiple partition data.

[0061] It should be noted that the execution subject of this embodiment is a data aggregation and processing device, which has functions such as data processing, data communication and program execution. The data aggregation and processing device can be an integrated controller, a control computer or other devices with similar functions. This embodiment does not limit the scope of the data aggregation and processing device.

[0062] It should be noted that the data aggregation processing method in this invention can be applied to a data aggregation processing system, for details of which can be found in [reference needed]. Figure 3 ,from Figure 3 As can be seen from the above, the system includes an upstream application, an intermediate processing layer, and a downstream application. The data aggregation processing method can be reflected in the data aggregation processing of the intermediate processing layer. Furthermore, the intermediate processing layer includes a data processing layer and a data aggregation layer. The data processing layer includes multiple processing nodes, and the data aggregation layer includes multiple aggregation nodes.

[0063] Understandably, the intermediate processing layer can process the raw business data of multiple upstream applications simultaneously. After the raw business data of each upstream application is processed by the data processing layer, the data aggregation layer aggregates the data and sends the aggregated data to the downstream application corresponding to that upstream application.

[0064] It should be understood that, based on Figure 3 It is known that the intermediate processing layer can process data from multiple upstream and downstream applications simultaneously. It partitions the upstream data, distributes the data of each partition to different first-level nodes, and then the second-level nodes aggregate the data from multiple partitions of the same upstream application and send the aggregated data to the downstream application.

[0065] It's worth noting that when using an intermediate application to process business data, upstream processes can generate massive amounts of data that are sent directly to the intermediate application, exceeding its processing capacity. This can lead to delays and data loss when data is sent downstream. Because an additional application layer is added, the entire business data transmission depends on the reliability of the intermediate application (which can be simply understood as the middle layer of the data aggregation and processing system). Failure to quickly detect and restore failed nodes in the intermediate application can also result in data delays and loss.

[0066] In specific implementation, such as Figure 3As shown, the entire system architecture is divided into two layers. The first layer is responsible for handling large volumes of raw business data. By using multiple nodes to obtain raw business data from different partitions of the distributed publish-subscribe messaging system (Apache Kafka, or Kafka for short), traffic is distributed, reducing the pressure on a single processing node. Each node in the first layer processes the raw business data, significantly reducing its volume. Then, the processed business data is sent to the second layer. A single node in the second layer receives all the business data, merges the distributed business data from the first layer according to appropriate rules, and sends the final correct business data downstream.

[0067] It's worth noting that the diagram only illustrates the scenario where a pair of upstream and downstream applications use this architecture to process business data. In reality, a single processing node can handle business data from multiple pairs of upstream and downstream applications simultaneously. However, to minimize the impact of a single node failure on business data transmission, a single node cannot simultaneously handle multiple topic partitions from a pair of upstream and downstream applications. Furthermore, Layer 2 can also have multiple processing nodes, and each node can handle multiple topics.

[0068] Understandably, the amount of original business data is very large, and the data in the multiple partitions after partitioning the original business data is a part of the original business data.

[0069] Step S20: Distribute the partition data to different first-level nodes, and process the partition data through the first-level nodes to obtain multiple reference business data.

[0070] Understandably, a layer of nodes includes multiple processing nodes, with each partition of data assigned to a corresponding processing node. It should be emphasized that, in order to minimize the impact of a single node failure on business data transmission, partitions of the same original business data are not assigned to the same processing node at the same time. This ensures that the failure of a single node will not affect the data processing of the same upstream application.

[0071] It should be emphasized that the same processing node can process data from different upstream applications simultaneously.

[0072] It should be understood that partitioning the original business data into multiple partitions can be done through Kafka. Kafka receives the original business data that needs to be processed, as shown in the reference. Figure 3 The original business data is divided into multiple partitions using Kafka, with each partition containing a portion of the original business data.

[0073] Understandably, reference business data can be... Figure 3 The data is processed by each processing node in the first layer of nodes.

[0074] It should be noted that the reference is... Figure 3 In a data aggregation and processing system, to achieve mutual backup between nodes at the same level and to quickly add a processing layer for specific business functions, the system needs to maintain information such as: the number of topics that a layer of processing nodes can process simultaneously (in the diagram, a topic represents the data transmission thread allocated by Kafka to an upstream application), all topic data that the layer needs to receive and process, and the number of partitions for each topic; the topic data that a layer of nodes is currently processing, and the corresponding heartbeat information, where the heartbeat information can be time information updated periodically by each node.

[0075] Step S30: Aggregate the multiple reference service data to the second-level node, and then aggregate the multiple reference service data through the second-level node to obtain aggregated data.

[0076] Understandably, the original business data of the same upstream application is divided into multiple processing nodes at the first-level node for processing. However, if the multiple partition data processed by multiple processing nodes at the first level belong to the same upstream application, they need to be sent to the same corresponding downstream application. After aggregating multiple reference business data, they are sent together to the downstream application corresponding to the upstream application.

[0077] It should be noted that the second-layer node includes multiple aggregation nodes. An aggregation node can process multiple pairs of upstream and downstream data simultaneously. Simply put, upstream application A is divided and processed by the first-layer node to obtain a1-a5, and upstream application B is divided and processed by the first-layer node to obtain b1-b3. a1-a5 and b1-b3 can be aggregated in the same aggregation node and then sent to the corresponding downstream applications A' and B'.

[0078] It should be noted that in order to achieve mutual backup of nodes at the same level and to quickly add second-level processing nodes for specific businesses, the system needs to perform information maintenance. The information that needs to be maintained includes the information of the second-level nodes, which includes the number of topics that the second-level nodes can process simultaneously (the number of upstream applications whose data has been processed by the first-level nodes can be processed at the same time), information on all topics that the second-level nodes need to receive and process, the data records that the second-level aggregation nodes are currently processing, and the heartbeat information of the aggregation nodes.

[0079] Step S40: Send the aggregated data to the downstream application.

[0080] Understandably, the aggregated data here is the data obtained after the upstream data is divided, processed by one layer of nodes, and then aggregated by two layers of nodes.

[0081] It should be emphasized that the aggregated data can include all the original business data. However, because the original business data is divided into multiple partitions, the same piece of data may appear in different first-level nodes and then be aggregated through second-level nodes. Simply put, the aggregated data may contain reference business data obtained from the same data processing. Reference business data obtained from the same data processing can be merged at the aggregation node.

[0082] This embodiment provides the ability to process ultra-large batches of business data through multiple first-layer nodes. It can also dynamically configure the number of processing nodes required by the corresponding business according to the different business data volume. It can quickly expand the processing capacity when the upstream data volume changes. Then, the data is aggregated through second-layer nodes to realize the forwarding of a large amount of raw business data from upstream, avoiding the problems of data delay and loss that are easy to occur when processing a large amount of data.

[0083] refer to Figure 4 , Figure 4 This is a flowchart illustrating a second embodiment of a data aggregation processing method according to the present invention.

[0084] Based on the first embodiment described above, the data aggregation processing method of this embodiment further includes, after step S20:

[0085] Step S21: Query the topic data processing table to obtain the data being processed and the total amount of data that can be processed by the target processing node. The target processing node is any one of the nodes in the first layer.

[0086] Understandably, the target processing node can be any processing node in a layer of nodes. It should be emphasized that only one target processing node can exist at any given time.

[0087] It should be understood that the total amount of data that the target processing node can process can be obtained by querying the topic data processing table to get the data that the target processing node is currently processing. The total amount of data that each node can process can be fixed, which can be understood as a pre-determined processing capacity.

[0088] Understandably, the topic data processing table can include the processing node corresponding to each topic partition and the heartbeat information of that node.

[0089] It should be noted that after a layer node starts processing data, it can periodically decide which business data should be received and processed, and quickly use new nodes to process the Topic partition data that was released due to anomalies.

[0090] Step S22: Determine whether the target processing node has spare data processing capacity based on the amount of data being processed and the total amount of data that can be processed by the target processing node.

[0091] It is understandable that each processing node can be regarded as a processor or a machine for processing data. Meanwhile, it is easy to understand that the maximum data processing capacity of each node is fixed when the node is generated.

[0092] It should be understood that the remaining data processing capacity can be simply understood as the amount of data that a processing node can still process.

[0093] It should be understood that subtracting the amount of data being processed by the target node from the total amount of data that the target node can process can obtain the remaining data processing capacity of the target node.

[0094] In a specific implementation, the data to be processed is A, and the total amount of data that can be processed is B. Then, compare the sizes of A and B. If A > B, randomly select A - B Topic data that the current node is processing for release, delete the processing records of the released Topic data on this node, and close the corresponding consumption thread (the consumption thread can be understood as the data processing plan of this node for processing this Topic data), release the Kafka client, and clean up the corresponding processing records; if A = B, directly end the process; if A < B, enter step S23. Starting from step S23, when multiple first-level nodes execute the corresponding logic, they are in a mutually exclusive relationship. Only one instance can execute the corresponding step at the same time, and other instances need to wait for the previous instance to complete before they can execute the corresponding step.

[0095] Step S23: When there is remaining data processing capacity in the target processing node, query the topic data table to be processed, obtain the data to be processed that meets the preset requirements in the topic data table to be processed, and allocate the data to be processed to the target processing node. The topic data table to be processed includes the data information of each topic, and the data information includes the data to be processed, the number of partitions, and the total number of processing nodes.

[0096] It is understandable that the topic data table to be processed can include the data to be processed of each topic in the data aggregation processing system, and this topic data table to be processed can be updated according to the real-time processing results of each processing node for the data.

[0097] It should be understood that the topic data table to be processed includes the real-time data to be processed by the upstream application, as well as the partitions of each topic and the total number of partitions of each topic.

[0098] It should be noted that when the target processing node has spare data processing capacity, the topic pending data table is queried to determine whether the total number of partitions of the topic pending data in the topic pending data table is greater than the total number of processing nodes of the topic pending data; if the number of partitions is greater than the total number of processing nodes, the topic data processing table is queried to determine whether the target processing node processes the partition data of the topic pending data; if the target processing node does not process the partition data of the topic pending data, the data pending data is allocated to the target processing node.

[0099] In this context, it is understandable that querying the topic pending data table to determine whether the total number of partitions of the topic corresponding to the real-time pending data is greater than the total number of processing nodes is to find data in the real-time data list where the total number of partitions is greater than the number of nodes. Simply put, when the business data of an upstream application is divided into multiple partitions, but there are not enough processing nodes to process each partition, the unprocessed data will be reassigned to data processing nodes with spare data processing capabilities.

[0100] It is important to emphasize that if there are not enough processing nodes to process the data in each partition, it is necessary to further distinguish between the data and the partition data that does not share the same topic with the processing nodes that currently have spare data processing capacity. In this case, the data should be assigned to the target processing node for data processing.

[0101] In the specific implementation summary, application A is divided into data partitions a1-a5, with 4 processing nodes corresponding to these partitions. Application B is divided into three data partitions b1-b3. At this point, processing node M, which processes partition b3 of application B, has spare data processing capacity. A query of the real-time data list reveals that the total number of partitions in application A (5) is greater than the number of processing nodes (4). Further querying the real-time data list shows that node M processes upstream applications B, K, and H, but does not process any partition data from application A. Therefore, the unprocessed data a5 is redistributed to node M as the target data.

[0102] It should be further explained that after allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the process also includes querying the topic data processing table and determining whether the topic of the current thread of the target node is a topic to be processed based on the query result. If the topic of the current thread of the target node is a topic to be processed, it is determined whether the topic processing record of the current thread of the target processing node exists. If it does not exist, the processing of the topic data to be processed by the current thread of the target processing node is stopped. If it exists, the node heartbeat of the target processing node is updated, and the total number of partitions of the topic where the target processing node is located is updated.

[0103] It is understood that the target processing node can be any processing node in a layer of nodes; it should be further understood that querying the topic data processing table and determining whether the topic of the target node's current thread is a topic to be processed based on the query results may be to check whether the target node's data to be processed is still in the topic data processing table. If it is not in the list, it may be that the processing task of the data has been cleared, and the target node no longer needs to process the data to be processed. If it is in the list, then the data to be processed continues to be processed.

[0104] It should be noted that determining whether the processing record of the pending data in the target processing node exists can be to determine whether the pending data still needs to be processed by the target processing node. The pending data in the target processing node can be manually deleted, after which the target node will no longer process the data. Alternatively, it can be because the target node stops updating its heartbeat due to its own failure and automatically releases the pending data. Releasing the pending data will automatically delete the processing record of the pending data in the target processing node.

[0105] It should be further explained that after allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the process further includes obtaining the heartbeat record of the target processing node, determining whether the target processing node has timed out based on the heartbeat record, deleting the processing record of the unprocessed data of the target processing node if the heartbeat of the target processing node has timed out, obtaining the total number of partitions of the upstream application corresponding to the target processing node, determining whether the total number of partitions is less than the total number of first-level nodes, and randomly deleting the data processing records of N target processing nodes if the total number of partitions is less than the total number of first-level nodes.

[0106] It is understandable that each processing node in a layer can periodically update its time, using that time as its heartbeat information.

[0107] Among them, it should be understood that the heartbeat timeout of the target processing node can be understood as that the target processing node has not processed data for a long time and has failed.

[0108] Among them, when the heartbeat information of the target processing node is not updated, it is considered that the target processing node cannot process data anymore. The data to be processed of the target processing node is written into the real-time data list, and the processing record of the data to be processed of the target processing node is deleted. Without a processing record, the data to be processed will not be processed again after the target processing node resumes heartbeat, avoiding multiple nodes from repeatedly processing the same data.

[0109] Among them, further in the specific implementation, query the total number of partitions A of all data to be processed, and obtain the number of processing nodes B of each upstream application. If it is found that A < B, randomly delete B - A processing records of the corresponding nodes.

[0110] It should be noted that the processing of each topic partition and the process in which the node discovers the data to be processed of the topic and starts processing are independent of each other and have no prior or subsequent association. After discovering the data to be processed of the topic and starting the processing thread, the subsequent processing process includes data aggregation as an independent thread.

[0111] In this embodiment, the data to be processed is dynamically allocated according to the processing capabilities of each processing node. When the data processing capacity of a node is excessive, it can process the data to be processed of other upstream and downstream nodes accordingly, avoiding the same node from processing the partition data of the same original data. Each first-level node regularly self-checks whether to continue processing the data being processed by each node, avoiding resource occupation caused by continuing to process data that has been revoked and data that has been released. At the same time, further regularly update the heartbeat of each node, and release the data to be processed of the node whose heartbeat record stops updating, avoiding data processing delays caused by the failure of a single node. When the number of nodes is greater than the total number of scores, it will cause waste of node data processing resources.

[0112] Reference Figure 5 , Figure 5 is a schematic flowchart of the second embodiment of a data aggregation processing method according to the present invention.

[0113] Based on the above first embodiment, after the step S30 of the data aggregation processing method in this embodiment, it further includes:

[0114] Step S31: Obtain the data being processed and the total amount of data that can be processed of the target aggregation node, and the target aggregation node is any one of the second-level nodes.

[0115] It should be understood that the total amount of data that can be processed by the target aggregation node can be obtained by querying the topic data aggregation table for the data being processed by the target aggregation node. The total amount of data that each node can process can be fixed, which can be understood as the processing capacity determined in advance.

[0116] Step S32: Determine whether the target aggregation node has free data aggregation capacity based on the data being processed by the target aggregation node and the total amount of data that can be processed.

[0117] It can be understood that each aggregation node can be regarded as a processor or a machine for processing data. At the same time, it is easy to understand that the maximum data aggregation capacity of each node is fixed when the node is generated.

[0118] It should be understood that the free data aggregation capacity can be simply understood as the amount of data that an aggregation node can still process.

[0119] It should be understood that subtracting the amount of data being processed by the target node from the total amount of data that can be aggregated by the target aggregation node can obtain the free data processing capacity of the target aggregation node.

[0120] In a specific implementation, the data to be processed by the target aggregation node is A, and the total amount of data that can be processed is B. Then, compare the sizes of A and B. If A > B, randomly select A - B Topic data being processed by the target aggregation node for release, delete the processing records of the released Topic data in the target aggregation node, and close the corresponding consumption threads to clean up the corresponding processing records; if A = B, directly end; if A < B, proceed to step S33. Starting from step S33, when multiple second-layer nodes execute the corresponding logic, they are in a mutually exclusive relationship. Only one instance can execute the corresponding step at the same time, and other instances need to wait for the previous instance to complete before they can execute the corresponding step.

[0121] Step S33: When the target aggregation node has free data aggregation capacity, query the topic data aggregation table to obtain the data to be aggregated that meets the preset requirements in the topic data aggregation table, and allocate the data to be aggregated to the target aggregation node. The topic data aggregation table includes the data information of each topic, and the data information includes the data to be aggregated and the total number of aggregation nodes.

[0122] It can be understood that each topic corresponds to an aggregation node. Simply put, the data of one topic is processed by multiple first-layer nodes and then aggregated to the same second-layer node.

[0123] Understandably, the topic data table to be aggregated can include the data to be aggregated from various topics in the data aggregation system, and this topic data table to be aggregated can be updated according to the real-time aggregation results of the data from each aggregation node.

[0124] It should be understood that the topic data table includes real-time data to be aggregated from upstream applications.

[0125] It should be noted that when the target aggregation node has spare data aggregation capabilities, the topic data to be aggregated is queried. The data to be aggregated that meets the preset requirements in the topic data aggregation table can be understood as topic data that meets the preset requirements and is not processed by other aggregation nodes. Then, the data to be aggregated is allocated to the target aggregation node.

[0126] It should be further explained that the process of allocating the partition data to different second-level nodes and aggregating the partition data through the second-level nodes to obtain multiple reference business data can be understood as allocating partition data of different topics that the same first-level node may process to the same second-level node.

[0127] After obtaining the reference business data, the topic to be aggregated data table can be queried. Based on the query results, it can be determined whether the topic of the current thread of the target aggregation node is a topic to be processed. If the topic of the current thread of the target node is a topic to be processed, it is determined whether the topic processing record of the current thread of the target aggregation node exists. If it does not exist, the processing of the topic to be processed of the current thread of the target processing node is stopped. If it exists, the node heartbeat of the target processing node is updated. The target aggregation node can be any node in the second-level nodes.

[0128] It should be further explained that after aggregating the multiple reference business data to the second-level node and aggregating the multiple reference business data through the second-level node to obtain aggregated data, the method further includes: obtaining the heartbeat record of the target aggregation node, determining whether the target aggregation node has timed out based on the heartbeat record; if the target aggregation node has timed out, deleting the processing record of the unprocessed data of the target aggregation node.

[0129] It is understandable that each aggregation node in the second layer can update the time periodically, using that node as its heartbeat information.

[0130] It should be understood that a target aggregation node's heartbeat timeout can be interpreted as the target aggregation node failing to perform data aggregation for an extended period, or as a problem occurring with the topic being processed by the target aggregation node.

[0131] Specifically, if the heartbeat information of the target aggregation node is not updated, the data to be aggregated for that target aggregation node is written to the real-time data list, and the aggregation record of the data to be aggregated for that target aggregation node is deleted. Without an aggregation record, the data to be aggregated will not be aggregated again after the target aggregation node resumes its heartbeat, thus avoiding multiple nodes repeatedly aggregating the same data.

[0132] It is understood that the target aggregation node can be any aggregation node in the second-level nodes; it should be further understood that querying the topic data aggregation table and determining whether the topic of the target node's current thread is the topic to be aggregated based on the query results may be to check whether the data to be aggregated of the target node is still in the topic data aggregation table. If it is not in the list, it may be that the aggregation task of the data has been cleared, and the target node does not need to continue to aggregate the data to be aggregated. If it is in the list, then the data to be aggregated will continue to be aggregated.

[0133] It should be noted that determining whether the aggregation record of the data to be aggregated in the target aggregation node exists can be to determine whether the data to be aggregated still needs to be aggregated by the target aggregation node. The data to be aggregated in the target aggregation node can be manually deleted. After deletion, the target node will no longer aggregate the data. Alternatively, the target node may automatically release the data to be aggregated after it stops updating its heartbeat due to its own failure. Releasing the data to be aggregated will automatically delete the aggregation record of the data to be aggregated in the target aggregation node.

[0134] It is important to emphasize that after aggregating data, the second-level aggregation node periodically checks for new, released data to be aggregated. If there is available data aggregation capacity, it further aggregates the new, released data. The main difference between second-level and first-level aggregation nodes is that second-level aggregation nodes do not need to consider the number of partitions. Only topics that have not been aggregated by any aggregation node can be written into the list of reference business data to be aggregated and thus retrieved.

[0135] It should be further emphasized that, during the process of aggregating business data, the second-level aggregation node performs thread self-checks similarly to the first-level aggregation node, but does not update the Topic partition count; at the same time, the second-level aggregation node will perform a scheduled task to check whether the heartbeat of the aggregation thread of all current aggregation nodes has timed out, but will not run the logic related to aggregation and partition count reduction.

[0136] In this embodiment, when the data aggregation capacity of each aggregation node is excessive, it can aggregate reference business data from other upstream and downstream nodes. At the same time, it periodically checks whether to continue processing the data being aggregated by each node, avoiding resource occupation due to continuing to aggregate data that has been revoked or released. Furthermore, it periodically updates the heartbeat of each node, and the heartbeat record stops updating the reference business data of the node and releases it, thus avoiding waste of node data aggregation resources.

[0137] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the data aggregation and processing device of the present invention.

[0138] like Figure 6 As shown, the data aggregation processing apparatus proposed in this embodiment of the invention includes:

[0139] The data processing module 10 is used to acquire the raw business data of each upstream application, partition the raw business data, and obtain multiple partition data.

[0140] The data processing module 10 is also used to allocate the partition data to different first-level nodes, and process the partition data through the first-level nodes to obtain multiple reference business data.

[0141] Data aggregation module 20 is used to aggregate the multiple reference business data to a second-level node, and to aggregate the multiple reference business data through the second-level node to obtain aggregated data;

[0142] The data aggregation module 20 is also used to send the aggregated data to downstream applications.

[0143] This embodiment provides the ability to process ultra-large batches of business data through multiple first-layer nodes. It can also dynamically configure the number of processing nodes required by the corresponding business according to the different business data volume. It can quickly expand the processing capacity when the upstream data volume changes. Then, the data is aggregated through second-layer nodes to realize the forwarding of a large amount of raw business data from upstream, avoiding the problems of data delay and loss that are easy to occur when processing a large amount of data.

[0144] In one embodiment, the data processing module 10 is further configured to query the topic data processing table to obtain the data being processed and the total amount of data that can be processed by the target processing node, wherein the target processing node is any one of the nodes in the first layer;

[0145] The presence of spare data processing capacity of the target processing node is determined based on the amount of data being processed and the total amount of data that can be processed by the target processing node.

[0146] When the target processing node has spare data processing capacity, the topic pending data table is queried to obtain pending data that meets the preset requirements. The pending data is then allocated to the target processing node. The topic pending data table includes data information for each topic, including pending data, number of partitions, and total number of processing nodes.

[0147] In one embodiment, the data processing module 10 is further configured to query the topic pending data table when the target processing node has spare data processing capacity, and determine whether the total number of partitions of the topic pending data in the topic pending data table is greater than the total number of processing nodes of the topic pending data.

[0148] If the number of partitions is greater than the total number of processing nodes, then query the topic data processing table to determine whether the target processing node processes the partition data of the topic of the data to be processed;

[0149] If the target processing node does not have partition data for the topic of the data to be processed, then the data to be processed will be assigned to the target processing node.

[0150] In one embodiment, the data processing module 10 is further configured to query the topic data processing table and determine whether the topic of the current thread of the target node is a topic to be processed based on the query result;

[0151] If the topic of the current thread of the target node is a topic to be processed, then determine whether the topic processing record of the current thread of the target processing node exists. If it does not exist, stop processing the topic data to be processed of the current thread of the target processing node.

[0152] If it exists, update the node heartbeat of the target processing node and update the total number of partitions of the topic where the target processing node is located.

[0153] In one embodiment, the data processing module 10 is further configured to acquire the heartbeat record of the target processing node and determine whether the heartbeat of the target processing node has timed out based on the heartbeat record;

[0154] If the heartbeat of the target processing node times out, the processing record of the unprocessed data of the target processing node is deleted;

[0155] Obtain the total number of partitions in the upstream application corresponding to the target processing node, and determine whether the total number of partitions is less than the total number of nodes in the first layer. If the total number of partitions is less than the total number of nodes in the first layer, randomly delete the data processing records of N target processing nodes.

[0156] In one embodiment, the data aggregation module 20 is further configured to obtain the data being processed and the total amount of data that can be processed by the target aggregation node, wherein the target aggregation node is any one of the second-layer nodes;

[0157] Based on the amount of data being processed and the total amount of data that the target aggregation node can process, determine whether the target aggregation node has spare data aggregation capabilities;

[0158] When the target processing node has spare data processing capacity, the topic data aggregation table is queried to obtain the data to be aggregated that meets the preset requirements. The data to be aggregated is then assigned to the target aggregation node. The topic data aggregation table includes data information for each topic, and the data information includes the data to be processed and the total number of processing nodes.

[0159] In one embodiment, the data aggregation module 20 is further configured to query the topic to be aggregated data table, and determine whether the topic of the current thread of the target aggregation node is a topic to be processed based on the query result;

[0160] If the topic of the current thread of the target node is a topic to be processed, then determine whether the topic processing record of the current thread of the target aggregation node exists. If it does not exist, stop processing the topic data to be processed of the current thread of the target processing node.

[0161] If it exists, update the node heartbeat of the target processing node, where the target aggregation node is any node in the second-level nodes.

[0162] In one embodiment, the data aggregation module 20 is further configured to acquire the heartbeat record of the target aggregation node and determine whether the heartbeat of the target aggregation node has timed out based on the heartbeat record;

[0163] If the heartbeat of the target aggregation node times out, the processing record of the pending data of the target aggregation node is deleted.

[0164] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0165] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0166] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0167] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0169] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0170] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data aggregation processing method, characterized in that, The data aggregation processing method includes: Obtain the raw business data from each upstream application, partition the raw business data, and obtain multiple partition data. The data of each partition is allocated to different first-level nodes, and the partition data is processed by the first-level nodes to obtain multiple reference business data. The multiple reference business data are aggregated to a second-level node, and the multiple reference business data are aggregated through the second-level node to obtain aggregated data; The aggregated data is sent to downstream applications; After allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the process further includes: Query the topic data processing table to obtain the data being processed and the total amount of data that can be processed by the target processing node. The target processing node is any one of the nodes in the first layer. The presence of spare data processing capacity of the target processing node is determined based on the amount of data being processed and the total amount of data that can be processed by the target processing node. When the target processing node has spare data processing capacity, query the topic pending data table to determine whether the number of partitions of the topic pending data in the topic pending data table is greater than the total number of processing nodes of the topic pending data. If the number of partitions is greater than the total number of processing nodes, then query the topic data processing table to determine whether the target processing node processes the partition data of the topic of the data to be processed; If the target processing node does not have partition data for the topic of the data to be processed, then the data to be processed will be assigned to the target processing node.

2. The data aggregation processing method as described in claim 1, characterized in that, After allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the process further includes: Query the topic data processing table, and determine whether the topic of the current thread of the target processing node is a topic to be processed based on the query results; If the topic of the current thread of the target processing node is a topic to be processed, then determine whether the topic processing record of the current thread of the target processing node exists. If it does not exist, stop processing the topic data to be processed of the current thread of the target processing node. If it exists, update the node heartbeat of the target processing node and update the total number of partitions of the topic where the target processing node is located.

3. The data aggregation processing method as described in claim 2, characterized in that, After allocating the partition data to different first-level nodes and processing the partition data through the first-level nodes to obtain multiple reference business data, the process further includes: Obtain the heartbeat record of the target processing node, and determine whether the heartbeat timeout of the target processing node is determined based on the heartbeat record; If the heartbeat of the target processing node times out, the processing record of the unprocessed data of the target processing node is deleted; Obtain the total number of partitions in the upstream application corresponding to the target processing node, and determine whether the total number of partitions is less than the total number of nodes in the first layer. If the total number of partitions is less than the total number of nodes in the first layer, randomly delete the data processing records of N target processing nodes.

4. The data aggregation processing method as described in claim 1, characterized in that, After aggregating the multiple reference service data to a Layer 2 node, and then aggregating the multiple reference service data through the Layer 2 node to obtain aggregated data, the process further includes: Obtain the data being processed and the total amount of data that can be processed by the target aggregation node, wherein the target aggregation node is any one of the second-level nodes; Based on the amount of data being processed and the total amount of data that the target aggregation node can process, determine whether the target aggregation node has spare data aggregation capabilities; When the target aggregation node has spare data aggregation capabilities, the topic data aggregation table is queried to obtain the data to be aggregated that meets the preset requirements. The data to be aggregated is then allocated to the target aggregation node. The topic data aggregation table includes data information for each topic, and the data information includes the data to be aggregated and the total number of aggregation nodes.

5. The data aggregation processing method as described in claim 4, characterized in that, After aggregating the multiple reference service data to a Layer 2 node, and then aggregating the multiple reference service data through the Layer 2 node to obtain aggregated data, the process further includes: Query the data table of topics to be aggregated, and determine whether the topic of the current thread of the target aggregation node is a topic to be processed based on the query results; If the topic of the current thread of the target aggregation node is a topic to be processed, then determine whether the topic processing record of the current thread of the target aggregation node exists. If it does not exist, stop processing the topic data to be processed of the current thread of the target aggregation node. If it exists, update the node heartbeat of the target aggregation node, where the target aggregation node is any node in the second-level nodes.

6. The data aggregation processing method as described in claim 5, characterized in that, After aggregating the multiple reference service data to a Layer 2 node, and then aggregating the multiple reference service data through the Layer 2 node to obtain aggregated data, the process further includes: Obtain the heartbeat record of the target aggregation node, and determine whether the heartbeat of the target aggregation node has timed out based on the heartbeat record; If the heartbeat of the target aggregation node times out, the processing record of the pending data of the target aggregation node is deleted.

7. A data aggregation processing apparatus, characterized in that, The data aggregation processing device includes: The data processing module is used to acquire raw business data from various upstream applications, partition the raw business data, and obtain multiple partition data. The data processing module is also used to allocate the data of each partition to different first-level nodes, and process the partition data through the first-level nodes to obtain multiple reference business data. The data aggregation module is used to aggregate the multiple reference business data to the second-level node, and then aggregate the multiple reference business data through the second-level node to obtain aggregated data; The data aggregation module is also used to send the aggregated data to downstream applications; The data processing module is also used to query the topic data processing table to obtain the data being processed and the total amount of data that can be processed by the target processing node. The target processing node is any one of the nodes in the first layer. The presence of spare data processing capacity of the target processing node is determined based on the amount of data being processed and the total amount of data that can be processed by the target processing node. When the target processing node has spare data processing capacity, query the topic pending data table to determine whether the number of partitions of the topic pending data in the topic pending data table is greater than the total number of processing nodes of the topic pending data. If the number of partitions is greater than the total number of processing nodes, then query the topic data processing table to determine whether the target processing node processes the partition data of the topic of the data to be processed; If the target processing node does not have partition data for the topic of the data to be processed, then the data to be processed will be assigned to the target processing node.

8. A data aggregation and processing device, characterized in that, The device includes: a memory, a processor, and a data aggregation processing program stored in the memory and executable on the processor, the data aggregation processing program being configured to implement the data aggregation processing method as described in any one of claims 1 to 6.

Images