Network data high-frequency synchronous processing method, device, equipment and medium

Abstract

The embodiment of the invention provides a network data high-frequency synchronous processing method and device, equipment and a storage medium. The method comprises the steps that data sent by terminal nodes needing data aggregation in the terminal nodes are received in batches through a center node; and for the received data of each batch of terminal nodes needing data aggregation, sending the data of the first part of terminal nodes to a target terminal node, and sending the data of the second part of terminal nodes to terminal nodes which do not send data to the central node, the terminal nodes which do not send data to the central node send the data of the terminal nodes and the received data of the second part of terminal nodes to the central node; according to the invention, the data of each terminal node can be received in a batch receiving mode, so that the problems of network congestion and overload processing capability of the central node caused by simultaneous inrush of a large amount of data can be avoided, and the whole data aggregation process can be effectively managed and regulated.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for high-frequency synchronous processing of network data. Background Art

[0002] With the continuous advancement of science and technology, the requirements for large-scale model computing are getting higher and higher. In the existing technology, when multiple nodes in a large model interact and converge with the central node for high-frequency data communication, tree algorithms and ring algorithms are generally used to achieve communication between multiple nodes. However, these two algorithms will face bandwidth bottlenecks and communication efficiency problems when facing large-scale nodes and central nodes for high-frequency data convergence communication. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a method, apparatus, device and medium for high-frequency synchronization processing of network data that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problem, an embodiment of the present invention discloses a method for high-frequency synchronization processing of network data, wherein the network includes a plurality of nodes, and the plurality of nodes include a central node and a terminal node; the method includes:

[0005] Receiving, by the central node, data sent by the terminal nodes that need to aggregate data in batches;

[0006] For each batch of received data from terminal nodes that need to be aggregated, the data from the first part of the terminal nodes are sent to the target terminal node, and the data from the second part of the terminal nodes are sent to the terminal nodes that have not sent data to the central node, so that the terminal nodes that have not sent data to the central node send their own data and the received data from the second part of the terminal nodes to the central node.

[0007] Optionally, the receiving, by the central node, data sent by the terminal nodes that need to perform data aggregation in batches includes:

[0008] Determine the maximum parallel quantity corresponding to the data bandwidth of the central node;

[0009] Determining the number of terminal nodes that need to perform data aggregation among the terminal nodes;

[0010] Determine the maximum number of nodes for each batch of terminal nodes that need to perform data aggregation according to the maximum parallel number and the number of terminal nodes that need to perform data aggregation;

[0011] Dividing the terminal nodes that need to perform data aggregation into multiple batches of terminal nodes according to the maximum number of nodes;

[0012] The data sent by the terminal nodes in the plurality of batches of terminal nodes are received in batches.

[0013] Optionally, the method further comprises:

[0014] Obtaining the data bandwidth of the central node;

[0015] Determining whether the data bandwidth is greater than a preset bandwidth;

[0016] The receiving, by the central node, data sent by the terminal nodes that need to aggregate data in batches includes:

[0017] If the data bandwidth is less than the preset bandwidth, the data sent by the terminal nodes that need to perform data aggregation among the terminal nodes are received in batches through the central node.

[0018] Optionally, the method further comprises:

[0019] If the data bandwidth value is greater than the preset bandwidth, the data sent by the terminal nodes that need to perform data aggregation among the terminal nodes is received through the central node, and the data sent by the terminal nodes that need to perform data aggregation is sent to the target terminal node.

[0020] Optionally, the method further comprises:

[0021] The central node receives in batches data processing results corresponding to the terminal nodes that need to disperse data, which are sent by the target terminal nodes;

[0022] The received data processing results corresponding to each batch of terminal nodes that need to disperse data are sent to the corresponding terminal nodes in batches.

[0023] Optionally, the receiving in batches by the central node the data processing results corresponding to the terminal nodes that need to disperse the data and that are sent by the target terminal nodes, includes:

[0024] Determining the number of terminal nodes that need to perform data dispersion;

[0025] Determine the maximum parallel quantity corresponding to the data bandwidth of the central node;

[0026] Determine the number of terminal nodes for each batch that need to be distributed according to the maximum parallel number and the number of terminal nodes that need to be distributed;

[0027] According to the number of terminal nodes in each batch that need to perform data dispersion, the data processing results corresponding to the terminal nodes that need to perform data dispersion and sent by the target terminal nodes are received in batches.

[0028] Optionally, sending the received data processing results corresponding to each batch of terminal nodes that need to disperse data to the corresponding terminal nodes in batches includes:

[0029] For each batch of received data processing results corresponding to the terminal nodes that need to be dispersed, the data processing results corresponding to the third part of the terminal nodes and the data processing results corresponding to the fourth part of the terminal nodes are sent to the third part of the terminal nodes;

[0030] And receiving the data processing results corresponding to the fourth part terminal nodes sent by the third part terminal nodes, and sending the data processing results corresponding to the fourth part terminal nodes to the fourth part terminal nodes.

[0031] The present invention also discloses a high-frequency synchronous processing device for network data, wherein the network includes a plurality of nodes, and the plurality of nodes include a central node and a terminal node; the device includes:

[0032] A first receiving module, configured to receive, through the central node, data sent by terminal nodes that need to aggregate data among the terminal nodes in batches;

[0033] The first sending module is used to send the data of the first part of the terminal nodes to the target terminal node, and send the data of the second part of the terminal nodes to the terminal nodes that have not sent data to the central node, so that the terminal nodes that have not sent data to the central node send their own data and the received data of the second part of the terminal nodes to the central node.

[0034] Optionally, the first receiving module includes:

[0035] A first determination submodule is used to determine the maximum parallel quantity corresponding to the data bandwidth of the central node;

[0036] A second determination submodule is used to determine the number of terminal nodes that need to perform data aggregation among the terminal nodes;

[0037] A third determination submodule is used to determine the maximum number of nodes of each batch of terminal nodes that need to perform data aggregation according to the maximum parallel number and the number of terminal nodes that need to perform data aggregation;

[0038] A division submodule, used for dividing the terminal nodes that need to perform data aggregation into multiple batches of terminal nodes according to the maximum number of nodes;

[0039] The first receiving submodule is used to receive data sent by terminal nodes in the multiple batches of terminal nodes in batches.

[0040] Optionally, the device further comprises:

[0041] An acquisition module, used to acquire the data bandwidth of the central node;

[0042] A judging module, used to judge whether the data bandwidth is greater than a preset bandwidth;

[0043] The first receiving module includes:

[0044] The second receiving submodule is configured to receive, in batches, data sent by terminal nodes that need to perform data aggregation among the terminal nodes through the central node if the data bandwidth is less than the preset bandwidth.

[0045] Optionally, the device further comprises:

[0046] The third receiving submodule is used for receiving data sent by the terminal nodes that need data aggregation among the terminal nodes through the central node if the data bandwidth value is greater than the preset bandwidth, and sending the data sent by the terminal nodes that need data aggregation to the target terminal node.

[0047] Optionally, the device further comprises:

[0048] A second receiving module is used to receive, in batches, data processing results corresponding to the terminal nodes that need to disperse data and sent by the target terminal nodes through the central node;

[0049] The second sending module is used to send the received data processing results corresponding to the batches of terminal nodes that need to disperse data to the corresponding terminal nodes in batches.

[0050] Optionally, the second receiving module includes:

[0051] A fourth determination submodule is used to determine the number of terminal nodes that need to perform data dispersion;

[0052] A fifth determination submodule, used to determine the maximum parallel quantity corresponding to the data bandwidth of the central node;

[0053] A sixth determination submodule, used to determine the number of terminal nodes that need to perform data dispersion in each batch according to the maximum parallel quantity and the number of terminal nodes that need to perform data dispersion;

[0054] The third receiving submodule is used to receive the data processing results corresponding to the terminal nodes that need to be dispersed and sent by the target terminal nodes in batches according to the number of the terminal nodes that need to be dispersed in each batch.

[0055] Optionally, the second sending module includes:

[0056] A sending submodule, for sending the data processing results corresponding to the third part of the terminal nodes and the data processing results corresponding to the fourth part of the terminal nodes to the third part of the terminal nodes according to the received data processing results corresponding to the batches of terminal nodes that need to be dispersed;

[0057] The fourth receiving submodule is used to receive the data processing results corresponding to the fourth part terminal nodes sent by the third part terminal nodes, and send the data processing results corresponding to the fourth part terminal nodes to the fourth part terminal nodes.

[0058] The present invention also discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the above-mentioned high-frequency synchronization processing method for network data when executed by the processor.

[0059] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned high-frequency synchronization processing method for network data are implemented.

[0060] The embodiments of the present invention include the following advantages:

[0061] The present invention discloses a method for high-frequency synchronous processing of network data. The present invention can receive data of each terminal node in a batch receiving manner, thereby avoiding network congestion caused by the simultaneous influx of a large amount of data and the problem of central node bandwidth bottleneck. The central node can perform subsequent processing on the received data in an orderly manner according to a certain order and rhythm, which is convenient for effective management and regulation of the entire data aggregation process. By sending a part of the data to the target terminal node and another part to the terminal node that has not sent the data, the pressure of data transmission can be dispersed, and each terminal node can also participate in the data transfer and further aggregation process, so that the traffic load distribution in the network is more balanced, which helps to improve the operation efficiency and stability of the entire network. In the embodiment of the present invention, the central node only plays the role of data forwarding, and does not need to process the data sent by multiple terminals, which ensures that the communication delay will not increase significantly. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flowchart of the steps of a method for high-frequency synchronization processing of network data provided by an embodiment of the present invention;

[0063] Figure 2 It is a schematic diagram of a data aggregation process provided by an embodiment of the present invention;

[0064] Figure 3 It is a schematic diagram of a data dispersion process provided by an embodiment of the present invention;

[0065] Figure 4 It is a structural block diagram of a network data high-frequency synchronization processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] With the continuous advancement of science and technology, the requirements for large model computing are getting higher and higher. In the existing technology, when multiple nodes in a large model interact and converge high-frequency data with a central node, a tree algorithm and a ring algorithm are generally used to realize communication between multiple nodes. The principles of the Tree algorithm and the Ring algorithm are as follows:

[0068] Ring algorithm: This algorithm has more communication steps. Data needs to be transmitted step by step along the entire ring, from the first node to the last node. The communication time increases linearly with the number of nodes. When the number of nodes is large, the communication efficiency of this algorithm is not high.

[0069] Tree algorithm: This algorithm divides the data set into different subsets through a series of tests on the characteristics of the data set, and finally obtains the decision result. It is like a tree-like flowchart, starting from the root node, branching according to different conditions until the leaf node obtains the classification or prediction result. However, this algorithm cannot use the sending and receiving data at the same time in its implementation. When facing multiple nodes for communication, the bandwidth of the central node will have a bottleneck problem, thus affecting the transmission efficiency.

[0070] One of the core concepts of the embodiments of the present invention is that the present invention can receive data from each terminal node in batches, thereby avoiding network congestion caused by the simultaneous influx of a large amount of data and the problem of central node bandwidth bottleneck. The central node can perform subsequent processing on the received data in an orderly manner according to a certain order and rhythm, which is convenient for effective management and regulation of the entire data aggregation process. By sending part of the data to the target terminal node and the other part to the terminal node that has not sent data, the pressure of data transmission can be dispersed, and each terminal node can also participate in the data transfer and further aggregation process, so that the traffic load distribution in the network is more balanced, which helps to improve the operating efficiency and stability of the entire network. In the embodiments of the present invention, the central node only plays the role of data forwarding and does not have to process the data sent by multiple terminals, which ensures that the communication delay will not increase significantly.

[0071] Reference Figure 1, shows a flowchart of a method for high-frequency synchronization processing of network data provided by an embodiment of the present invention, wherein the network includes a plurality of nodes, and the plurality of nodes include a central node and a terminal node; the method may include the following steps:

[0072] Step 101: receiving data sent by terminal nodes that need to aggregate data in batches through a central node.

[0073] In an embodiment of the present invention, the method can be applied to a VRB system. The VRB (V2V RDMA Bandwidth, remote direct memory access bandwidth based on visual networking) system is an architecture for network data transmission and organization. It is a system related to virtual resource allocation and data grouping, and is used to efficiently process data aggregation and transmission in a network environment including a central node and terminal nodes. In the VRB system network environment, there are a central node and multiple terminal nodes, some of which have the need to aggregate data, that is, to transmit their own data to a specific location for integration processing.

[0074] The terminal nodes can prepare the data to be sent according to established rules or protocols. These data may cover various types of information, such as environmental data collected by sensors, business operation records of the user end, etc. The central node will set certain batch standards, such as according to time intervals, according to the regional grouping of terminal nodes, or according to the amount of data, to determine how to receive in batches. The specific method of batching is not limited here.

[0075] Based on the requirements of the central node, the terminal node actively initiates a data transmission request to the central node when the corresponding batch time or conditions are met. The central node opens the corresponding receiving port or service to receive this data. When transmitting each batch of data, it will follow the network communication protocol and gradually transmit the data from the terminal node to the central node through the network link. In this process, the central node only plays the role of receiving and forwarding data.

[0076] Step 102, for each batch of received terminal node data that needs to be aggregated, the data of the first part of the terminal nodes are sent to the target terminal node, and the data of the second part of the terminal nodes are sent to the terminal nodes that have not sent data to the central node, so that the terminal nodes that have not sent data to the central node send their own data and the received data of the second part of the terminal nodes to the central node.

[0077] In an embodiment of the present invention, after the central node receives each batch of data, the terminal nodes to which the data belong are classified according to a pre-set strategy, and the terminal nodes are divided into a first part and a second part. The classification strategy can be based on various factors, such as the performance difference of the terminal nodes (high-performance terminal nodes are the first part, and low-performance terminal nodes are the second part), the network location of the terminal nodes (those close to the central node are the first part, and those farther away are the second part), or the importance level of the data (the terminal nodes to which important data belong are the first part, and the terminal nodes to which relatively less important data belong are the second part), etc.

[0078] After the central node determines the classification, it uses the network communication function to send the data of the first part of the terminal nodes to the target terminal node accurately according to the corresponding network address and other information. For the data of the second part of the terminal nodes, it is sent to those terminal nodes that have not sent data to the central node. This sending process also follows the network communication protocol to ensure that the data can accurately reach the corresponding receiving terminal node.

[0079] After receiving the data from the second part of the terminal nodes, the terminal nodes that have not sent data to the central node will start their own data integration function, merge and organize the data they originally wanted to send and the received external data. Then, these terminal nodes will initiate data transmission to the central node again according to the rules set by the network, and send the merged data back to the central node. The central node will then send the received data to the target terminal node for processing. By allowing the terminal nodes to participate in the transfer and interaction of data, the load pressure of data transmission in the network can be effectively dispersed, and performance bottlenecks in the central node can be avoided. At the same time, the redundant backup and transmission paths of data in the network are increased, which improves the overall success rate of data aggregation and the ability to cope with network failures, making the data aggregation process of the entire network more flexible, efficient and reliable.

[0080] The present invention discloses a method for high-frequency synchronous processing of network data. The present invention can receive data of each terminal node in a batch receiving manner, thereby avoiding network congestion caused by a large amount of data pouring in at the same time and the problem of bandwidth bottleneck in the central node. The central node can perform subsequent processing on the received data in an orderly manner according to a certain order and rhythm, which is convenient for effective management and regulation of the entire data aggregation process. By sending a part of the data to the target terminal node and the other part to the terminal node that has not sent the data, the pressure of data transmission can be dispersed, and the central node can be avoided from becoming the bottleneck of the data transmission of the entire network. Each terminal node can also participate in the data transfer and further aggregation process, so that the traffic load distribution in the network is more balanced, which helps to improve the operation efficiency and stability of the entire network.

[0081] In one embodiment of the present invention, data sent by terminal nodes that need to perform data aggregation among terminal nodes are received in batches by a central node, including: determining a maximum parallel number corresponding to the data bandwidth of the central node; determining the number of terminal nodes that need to perform data aggregation among terminal nodes; determining a maximum number of nodes for each batch of terminal nodes that need to perform data aggregation based on the maximum parallel number and the number of terminal nodes that need to perform data aggregation; dividing the terminal nodes that need to perform data aggregation into multiple batches of terminal nodes based on the maximum number of nodes; and receiving data sent by terminal nodes in multiple batches of terminal nodes in batches.

[0082] In the embodiment of the present invention, the data bandwidth of the central node refers to the amount of data that can be transmitted per unit time. This bandwidth resource is limited, and the maximum parallel number refers to the maximum number of terminal nodes that can simultaneously perform data receiving operations without exceeding the bandwidth carrying capacity. The bandwidth of the central node can be tested by a professional network performance testing tool to simulate a scenario in which multiple terminal nodes simultaneously send data to it, gradually increase the number of concurrent connections, and observe various indicators of data transmission, such as transmission delay, packet loss rate, etc. When these indicators begin to exceed a reasonable range, the corresponding number of concurrent connections at this time is the maximum parallel number corresponding to the data bandwidth of the central node.

[0083] Among the numerous terminal nodes in the entire network, not all terminal nodes have the need for data aggregation at any time. For example, some terminal nodes may be in standby mode and have not generated new data, or their data does not need to be sent to the central node for integration and processing temporarily. Therefore, it is necessary to determine which terminal nodes have the need for data aggregation based on business rules, pre-set conditions, etc. These conditions may involve factors such as data update time and event triggering.

[0084] By counting the terminal nodes that meet the above data aggregation demand judgment criteria one by one, the number of terminal nodes that need data aggregation can be accurately obtained. This number statistics can be assisted by the network management system. The central node sends a query request to each terminal node, and the terminal node responds according to its own situation. The central node then summarizes and counts the corresponding number.

[0085] Based on the data bandwidth limitation of the central node and the number of terminal nodes that actually need to aggregate data, in order to ensure that data can be received in batches smoothly and orderly, it is necessary to reasonably determine the maximum number of terminal nodes that can participate in data transmission in each batch, so that the central node will not exceed its own bandwidth carrying capacity due to receiving too much data from terminal nodes when receiving data in each batch. At the same time, it can also efficiently complete the data reception work for all terminal nodes that need to aggregate data, avoiding the reception process being too procrastinated and lengthy.

[0086] Based on the calculated maximum number of terminal nodes in each batch that need to aggregate data, all terminal nodes that need to aggregate data are grouped. They can be divided in sequence according to the terminal node identifiers, and each Q terminal node is divided into a batch until all terminal nodes that need to aggregate data are divided. For example, if the maximum number of nodes Q = 10, and the terminal nodes are numbered from 1 to 100 and need to aggregate data, then the terminal nodes numbered 1-10 can be divided into the first batch, 11-20 can be divided into the second batch, and so on, eventually forming multiple batches of terminal nodes.

[0087] After the terminal nodes are divided into batches, each batch of terminal nodes will initiate data sending requests to the central node in turn according to the pre-set time sequence or the scheduling arrangement of the central node. The central node starts the corresponding receiving service and receives data from each terminal node through the network link according to the network communication protocol.

[0088] During the entire batch receiving process, the central node will continue to monitor its own network status, bandwidth usage and other indicators. If any abnormal situation is found, the central node can take corresponding measures to make adjustments in time, such as suspending the reception of subsequent batches of data, first checking the problem of data transmission in the current batch, or dynamically adjusting the maximum number of nodes in each batch, such as appropriately reducing or increasing the number of batches, so as to better adapt to real-time changes in the network and ensure that data can be smoothly converged.

[0089] In one embodiment of the present invention, the method also includes: obtaining the data bandwidth of the central node; determining whether the data bandwidth is greater than a preset bandwidth; and receiving in batches through the central node data sent by terminal nodes that need to perform data aggregation among the terminal nodes, including: if the data bandwidth is less than the preset bandwidth, receiving in batches through the central node data sent by terminal nodes that need to perform data aggregation among the terminal nodes.

[0090] In the embodiments of the present invention, the data bandwidth of the central node can usually be obtained with the help of professional network monitoring tools. These tools can detect the relevant parameters of the network link where the central node is located in real time. For example, network bandwidth testing software can be used to send specific test data packets to the central node and monitor indicators such as the transmission rate of the data packets to determine its data bandwidth.

[0091] The preset bandwidth is a reference value that is set in advance based on factors such as the overall network planning and past data transmission experience. For example, it can be determined based on the data transmission efficiency expected to be achieved in the daily operation of the network, the minimum bandwidth requirements required to ensure stable transmission in similar business scenarios in the past, etc. It can be a fixed value. For example, for a data aggregation network within an enterprise, based on the data traffic analysis during past business peaks, the preset bandwidth is determined to be 100Mbps, which is used as a benchmark to measure whether the current central node bandwidth meets the demand.

[0092] When the data bandwidth is less than the preset bandwidth, it means that the central node does not have relatively sufficient bandwidth resources to cope with the data transmission tasks of the terminal nodes. At this time, the batch receiving method can be adopted to organize the terminal nodes to send and receive data in an orderly manner, avoiding the instantaneous network congestion that may be caused by a large number of terminal nodes transmitting data at the same time, and ensuring the accuracy and completeness of data aggregation.

[0093] In one embodiment of the present invention, the method further includes: if the data bandwidth value is greater than the preset bandwidth, receiving data sent by the terminal nodes that need to perform data aggregation among the terminal nodes through the central node, and sending the data sent by the terminal nodes that need to perform data aggregation to the target terminal node.

[0094] In an embodiment of the present invention, when the data bandwidth is greater than the preset bandwidth, it means that the central node has relatively sufficient bandwidth resources to cope with the data transmission tasks of the terminal nodes. At this time, the data of multiple terminal nodes that need to perform data aggregation can be directly received simultaneously through the central node. For example, the number of terminal nodes that need to perform data aggregation is 6, and the maximum parallel number of central nodes is 8. At this time, the data of the 6 terminal nodes can be sent to the central node at the same time, and then the central node sends the data of the 6 nodes to the target terminal node.

[0095] In one embodiment of the present invention, the method also includes: receiving in batches through a central node the data processing results corresponding to the terminal nodes that need to perform data dispersion and sent by the target terminal nodes; and sending the received data processing results corresponding to the terminal nodes that need to perform data dispersion in batches to the corresponding terminal nodes.

[0096] In the embodiment of the present invention, in the overall process of network data processing, the data has previously gone through a process of aggregating data from terminal nodes to central nodes and then forwarding it to target terminal nodes for corresponding processing. These target terminal nodes have certain data processing capabilities. After analyzing and calculating the received data, corresponding data processing results will be generated. Now these processing results need to be transmitted back to the initial part of the terminal nodes, that is, the terminal nodes that need to disperse the data. In order to ensure the orderliness of the receiving process and avoid problems such as excessive network transmission pressure, batch receiving is adopted for processing.

[0097] The target terminal node initiates a request to send the data processing results to the central node according to the agreed batch arrangement, and the central node receives this data in batches through the network link.

[0098] The central node needs to know exactly which corresponding terminal nodes each batch of data processing results should be sent to. This can be determined based on pre-set data flow rules and association relationships. For example, when data is initially aggregated and distributed, it is recorded which terminal nodes' data is sent to specific target terminal nodes for processing. Now, according to this correspondence, the data processing results returned by the target terminal nodes must be accurately sent back to those original terminal nodes. This correspondence can be stored and managed by establishing data mapping tables, configuration files, etc., to facilitate quick query and positioning by the central node when distributing data.

[0099] like Figure 2 , shows a schematic diagram of a data aggregation process provided by an embodiment of the present invention, there are 8 terminal nodes that need to perform data aggregation, node 1, node 2, node 5, and node 6 are the first batch of terminal nodes, node 3 and node 7 are the second batch of terminal nodes, and node 4 and node 8 are the third batch of terminal nodes. The sending order of the first batch of terminal nodes is earlier than that of the second terminal nodes, and the sending order of the second terminal nodes is earlier than that of the third terminal nodes. Node 1 can be forwarded to node 8 through the central node, node 2 can be forwarded to node 3 through the central node, node 5 can be forwarded to node 4 through the central node, and node 6 can be forwarded to node 7 through the central node; further, node 3 can forward its own data and node 2's data to node 4 through the central node, and node 7 can forward its own data and node 6's data to node 8 through the central node; further, node 4 can forward node 5's data, its own data, node 3's data, and node 2's data to node 8 through the central node, thereby completing data aggregation.

[0100] In one embodiment of the present invention, data processing results corresponding to terminal nodes that need data dispersion and sent by target terminal nodes are received in batches by a central node, including: determining the number of terminal nodes that need data dispersion; determining the maximum parallel number corresponding to the data bandwidth of the central node; determining the number of terminal nodes that need data dispersion in each batch according to the maximum parallel number and the number of terminal nodes that need data dispersion; and receiving data processing results corresponding to terminal nodes that need data dispersion and sent by target terminal nodes in batches according to the number of terminal nodes that need data dispersion in each batch.

[0101] In the embodiment of the present invention, there are many terminal nodes in the entire network data processing system, and some of the terminal nodes need to receive corresponding data processing results to continue subsequent business operations or further data processing after the previous data flow and processing process. These terminal nodes are the terminal nodes that need to disperse data. Accurately determining their number is the basis for the subsequent reasonable arrangement of data reception work, which helps to control the scale of data backhaul and distribution as a whole, and avoid unreasonable use of network resources caused by blindly receiving data and various data transmission problems that may arise.

[0102] Based on the receiving capacity limit of the central node, that is, the maximum number of parallel nodes and the number of terminal nodes that actually need to receive the data processing results, the number of terminal nodes participating in data reception in each batch is determined through reasonable calculation. The purpose is to ensure that in the entire data reception process, the bandwidth resources of the central node can be fully utilized, and the network will not be congested or the central node will not be unable to handle it due to receiving too much data from terminal nodes at the same time, thereby achieving smooth and orderly data reception and improving the efficiency and reliability of the entire data reception link.

[0103] The central node can make preparations for receiving in advance based on the calculated number of terminal nodes in each batch that need to be dispersed. This includes opening the corresponding receiving service and port, and configuring relevant parameters according to the network communication protocol, such as setting appropriate IP address binding, port monitoring range, etc., to ensure that the data packets of data processing results from the target terminal nodes can be accurately identified and accepted. At the same time, the central node will also record the terminal node information corresponding to each batch and the approximate scale of the expected data processing results, etc., for subsequent verification and management. For example, the number of terminal nodes that need to be dispersed is 25, the maximum parallel number of central nodes is 10, and the number of terminal nodes that need to be dispersed in each batch is 10, 10, and 5. Therefore, the central node can receive the data processing results in three times.

[0104] In one embodiment of the present invention, the data processing results corresponding to the received batches of terminal nodes that need to perform data dispersion are sent to the corresponding terminal nodes in batches, including: for the received batches of data processing results corresponding to the terminal nodes that need to perform data dispersion, the data processing results corresponding to the third part terminal nodes and the data processing results corresponding to the fourth part terminal nodes are sent to the third part terminal nodes; and the data processing results corresponding to the fourth part terminal nodes sent by the third part terminal nodes are received, and the data processing results corresponding to the fourth part terminal nodes are sent to the fourth part terminal nodes.

[0105] In the embodiment of the present invention, first, it is necessary to clarify how to distinguish the third part terminal nodes from the fourth part terminal nodes. This distinction is usually made based on pre-set rules or network data processing business logic. For example, the third part terminal nodes and the fourth part terminal nodes can be divided according to factors such as the geographical location of the terminal nodes (terminal nodes in different areas are divided into different parts), the performance difference of the terminal nodes (high performance as one part, relatively low performance as another part), and the type of data processing results (different business-related data processing results correspond to terminal nodes in different parts).

[0106] After determining the division, the central node extracts the data content corresponding to the third part terminal nodes and the fourth part terminal nodes from the batches of data processing results received. After receiving the data processing results corresponding to itself and the fourth part terminal nodes sent by the central node, the third part terminal node will perform corresponding processing according to the established business logic. Due to certain business needs or data processing flow requirements, the third part terminal node needs to transmit the data processing results corresponding to the fourth part terminal node back to the central node.

[0107] After successfully receiving the data processing results corresponding to the fourth part terminal node sent by the third part terminal node, the central node will organize and encapsulate the data again, and send these data processing results to the fourth part terminal node accurately according to the network address and other information of the fourth part terminal node.

[0108] like Figure 3, showing a schematic diagram of a data dispersion process provided by an embodiment of the present invention. Node 8 needs to disperse the data processing results to other terminal nodes. The nodes can be divided into three batches. The first batch is that node 8 sends the data processing results to node 4 through the central node. The second batch is: node 8 forwards the data processing results to node 7 through the central node, and node 4 forwards the data processing results to node 3 through the central node; the third batch is that node 8 forwards the data processing results to node 1 through the central node, node 4 forwards the data processing results to node 5 through the central node, node 3 forwards the data processing results to node 2 through the central node, and node 7 forwards the data processing results to node 6 through the central node.

[0109] The present invention discloses a method for high-frequency synchronous processing of network data. The present invention can receive data of each terminal node in batches, thereby avoiding network congestion caused by the simultaneous influx of a large amount of data and the overload of the central node processing capacity. The central node can perform subsequent processing on the received data in an orderly manner according to a certain order and rhythm, which is convenient for effective management and regulation of the entire data aggregation process. By sending a part of the data to the target terminal node and the other part to the terminal node that has not sent the data, the pressure of data transmission can be dispersed, and the central node can be avoided from becoming the bottleneck of the data transmission of the entire network. Each terminal node can also participate in the transfer and further aggregation of data, so that the traffic load distribution in the network is more balanced, which helps to improve the operation efficiency and stability of the entire network.

[0110] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0111] Reference Figure 4 , shows a structural block diagram of a network data high-frequency synchronization processing device provided by an embodiment of the present invention, the network includes multiple nodes, the multiple nodes include a central node and a terminal node; the device may include the following modules:

[0112] The first receiving module 201 is used to receive data sent by terminal nodes that need to perform data aggregation in batches through the central node;

[0113] The first sending module 202 is used to send the data of the first part of the terminal nodes to the target terminal node, and send the data of the second part of the terminal nodes to the terminal nodes that have not sent data to the central node, so that the terminal nodes that have not sent data to the central node send their own data and the received data of the second part of the terminal nodes to the central node.

[0114] The present invention discloses a high-frequency synchronous processing device for network data. The present invention can receive data from each terminal node in batches, thereby avoiding network congestion caused by a large amount of data pouring in at the same time and overloading the processing capacity of the central node. The central node can perform subsequent processing on the received data in an orderly manner according to a certain order and rhythm, which is convenient for effective management and regulation of the entire data aggregation process. By sending a part of the data to the target terminal node and the other part to the terminal node that has not sent the data, the pressure of data transmission can be dispersed, and the central node can be avoided from becoming the bottleneck of the data transmission of the entire network. Each terminal node can also participate in the data transfer and further aggregation process, so that the traffic load distribution in the network is more balanced, which helps to improve the operation efficiency and stability of the entire network.

[0115] In one embodiment of the present invention, the first receiving module includes:

[0116] A first determination submodule is used to determine the maximum parallel quantity corresponding to the data bandwidth of the central node;

[0117] The second determination submodule is used to determine the number of terminal nodes that need to perform data aggregation among the terminal nodes;

[0118] The third determination submodule is used to determine the maximum number of nodes of each batch of terminal nodes that need to perform data aggregation according to the maximum parallel number and the number of terminal nodes that need to perform data aggregation;

[0119] A division submodule is used to divide the terminal nodes that need to perform data aggregation into multiple batches of terminal nodes according to the maximum number of nodes;

[0120] The first receiving submodule is used to receive data sent by terminal nodes in multiple batches of terminal nodes in batches.

[0121] In one embodiment of the present invention, the device further includes:

[0122] An acquisition module, used to acquire the data bandwidth of the central node;

[0123] A judgment module, used to judge whether the data bandwidth is greater than a preset bandwidth;

[0124] The first receiving module comprises:

[0125] The second receiving submodule is configured to receive data sent by terminal nodes that need data aggregation in batches through the central node if the data bandwidth is less than a preset bandwidth.

[0126] In one embodiment of the present invention, the device further includes:

[0127] The third receiving submodule is used for receiving data sent by the terminal nodes that need data aggregation among the terminal nodes through the central node if the data bandwidth value is greater than the preset bandwidth, and sending the data sent by the terminal nodes that need data aggregation to the target terminal node.

[0128] In one embodiment of the present invention, the device further includes:

[0129] The second receiving module is used to receive data processing results corresponding to the terminal nodes that need to disperse data sent by the target terminal nodes in batches through the central node;

[0130] The second sending module is used to send the received data processing results corresponding to the batches of terminal nodes that need to disperse data to the corresponding terminal nodes in batches.

[0131] In one embodiment of the present invention, the second receiving module includes:

[0132] A fourth determination submodule is used to determine the number of terminal nodes that need to perform data dispersion;

[0133] A fifth determination submodule is used to determine the maximum parallel quantity corresponding to the data bandwidth of the central node;

[0134] A sixth determination submodule, used to determine the number of terminal nodes that need to perform data dispersion in each batch according to the maximum parallel quantity and the number of terminal nodes that need to perform data dispersion;

[0135] The third receiving submodule is used to receive the data processing results corresponding to the terminal nodes that need to be dispersed and sent by the target terminal nodes in batches according to the number of the terminal nodes that need to be dispersed in each batch.

[0136] In one embodiment of the present invention, the second sending module includes:

[0137] A sending submodule, for sending the data processing results corresponding to the terminal nodes in the third part and the data processing results corresponding to the terminal nodes in the fourth part to the terminal nodes in the third part according to the received data processing results corresponding to the terminal nodes in each batch that need to be dispersed;

[0138] The fourth receiving submodule is used to receive the data processing results corresponding to the fourth part terminal nodes sent by the third part terminal nodes, and send the data processing results corresponding to the fourth part terminal nodes to the fourth part terminal nodes.

[0139] The present invention discloses a high-frequency synchronous processing device for network data. The present invention can receive data from each terminal node in batches, thereby avoiding network congestion caused by a large amount of data pouring in at the same time and overloading the processing capacity of the central node. The central node can perform subsequent processing on the received data in an orderly manner according to a certain order and rhythm, which is convenient for effective management and regulation of the entire data aggregation process. By sending a part of the data to the target terminal node and the other part to the terminal node that has not sent the data, the pressure of data transmission can be dispersed, and the central node can be avoided from becoming the bottleneck of the data transmission of the entire network. Each terminal node can also participate in the data transfer and further aggregation process, so that the traffic load distribution in the network is more balanced, which helps to improve the operation efficiency and stability of the entire network.

[0140] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] An embodiment of the present invention further provides an electronic device, including:

[0142] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned network data high-frequency synchronization processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned network data high-frequency synchronization processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0144] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0145] It should be understood by those skilled in the art that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0146] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0150] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0151] The above is a detailed introduction to a method, device, equipment and storage medium for high-frequency synchronization processing of network data provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for high-frequency synchronous processing of network data, characterized in that: The network includes a plurality of nodes, and the plurality of nodes include a central node and a terminal node; the method includes: Receiving, by the central node, data sent by the terminal nodes that need to aggregate data in batches; For each batch of received data from terminal nodes that need to be aggregated, the data from the first part of the terminal nodes are sent to the target terminal node, and the data from the second part of the terminal nodes are sent to the terminal nodes that have not sent data to the central node, so that the terminal nodes that have not sent data to the central node send their own data and the received data from the second part of the terminal nodes to the central node.

2. The method according to claim 1, characterized in that The receiving, by the central node, data sent by the terminal nodes that need to aggregate data in batches includes: Determine the maximum parallel quantity corresponding to the data bandwidth of the central node; Determining the number of terminal nodes that need to perform data aggregation among the terminal nodes; Determine the maximum number of nodes for each batch of terminal nodes that need to perform data aggregation according to the maximum parallel number and the number of terminal nodes that need to perform data aggregation; Dividing the terminal nodes that need to perform data aggregation into multiple batches of terminal nodes according to the maximum number of nodes; The data sent by the terminal nodes in the plurality of batches of terminal nodes are received in batches.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining the data bandwidth of the central node; Determining whether the data bandwidth is greater than a preset bandwidth; The receiving, by the central node, data sent by the terminal nodes that need to aggregate data in batches includes: If the data bandwidth is less than the preset bandwidth, the data sent by the terminal nodes that need to perform data aggregation among the terminal nodes are received in batches through the central node.

4. The method according to claim 3, characterized in that The method further comprises: If the data bandwidth value is greater than the preset bandwidth, the data sent by the terminal nodes that need to perform data aggregation among the terminal nodes is received through the central node, and the data sent by the terminal nodes that need to perform data aggregation is sent to the target terminal node.

5. The method according to claim 1, characterized in that The method further comprises: The central node receives in batches data processing results corresponding to the terminal nodes that need to disperse data, which are sent by the target terminal nodes; The received data processing results corresponding to each batch of terminal nodes that need to disperse data are sent to the corresponding terminal nodes in batches.

6. The method according to claim 5, characterized in that The data processing results corresponding to the terminal nodes that need to disperse data and are sent by the target terminal nodes and received in batches by the central node include: Determining the number of terminal nodes that need to perform data dispersion; Determine the maximum parallel quantity corresponding to the data bandwidth of the central node; Determine the number of terminal nodes for each batch that need to be distributed according to the maximum parallel number and the number of terminal nodes that need to be distributed; According to the number of terminal nodes in each batch that need to perform data dispersion, the data processing results corresponding to the terminal nodes that need to perform data dispersion and sent by the target terminal nodes are received in batches.

7. The method according to claim 5, characterized in that The step of sending the received data processing results corresponding to each batch of terminal nodes that need to disperse data to the corresponding terminal nodes in batches includes: For each batch of received data processing results corresponding to the terminal nodes that need to be dispersed, the data processing results corresponding to the third part of the terminal nodes and the data processing results corresponding to the fourth part of the terminal nodes are sent to the third part of the terminal nodes; And receiving the data processing results corresponding to the fourth part terminal nodes sent by the third part terminal nodes, and sending the data processing results corresponding to the fourth part terminal nodes to the fourth part terminal nodes.

8. A network data high-frequency synchronization processing device, characterized in that: The network includes a plurality of nodes, wherein the plurality of nodes include a central node and a terminal node; the device includes: A first receiving module, configured to receive, through the central node, data sent by terminal nodes that need to aggregate data among the terminal nodes in batches; The first sending module is used to send the data of the first part of the terminal nodes to the target terminal node, and send the data of the second part of the terminal nodes to the terminal nodes that have not sent data to the central node, so that the terminal nodes that have not sent data to the central node send their own data and the received data of the second part of the terminal nodes to the central node.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the high-frequency synchronization processing method for network data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for high-frequency synchronization processing of network data as described in any one of claims 1 to 7 are implemented.

Images