Data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: combining all to-be-combined nodes into at least two tree structures, and connecting root nodes of the at least two tree structures into an annular structure. A plurality of tree structures process data in parallel, and compared with a single tree structure, centralized transmission of the data in a small number of high-level child nodes is reduced. The annular structure is connected with the root nodes, so that when the summarized data is transmitted among the root nodes, data transmission paths are more dispersed, the transmission bottleneck of high-level nodes in a single tree structure is avoided, and the overall bandwidth utilization rate of all the nodes is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device, an electronic device and a storage medium. Background Art

[0002] AllReduce is an operation that reduces the data on all nodes in a distributed system and broadcasts the reduced results to all nodes so that all nodes eventually have the same reduced data.

[0003] In the relevant technology of data processing, all nodes are usually combined into a tree structure, wherein the tree structure includes a root node and multiple child nodes. The child nodes summarize their own data to the root node, and the root node distributes the summarized data to each child node. Since the tree structure is hierarchical, the more child nodes at the bottom layer and the higher the layer, the fewer child nodes there are. The number of bytes at the highest layer is two, and the root node is above it. When data is transmitted from the child node to the root node, as the layer increases, the overall bandwidth utilization of all nodes gradually decreases, resulting in a low overall bandwidth utilization of all nodes. Summary of the invention

[0004] The present application provides a data processing method and device, an electronic device and a storage medium to at least solve the problem of low overall bandwidth utilization of all nodes in the related art.

[0005] This application provides a data processing method, including:

[0006] According to the preset combination algorithm, all nodes to be combined are combined into at least two tree structures;

[0007] Connecting the root nodes of at least two tree structures into a ring structure;

[0008] Aggregate the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form the summary data corresponding to each tree structure;

[0009] Pass the summary data of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure;

[0010] The summary data is distributed to all the child nodes of each tree structure using the root node of each tree structure, so that all the child nodes of each tree structure can process the summary data.

[0011] The present application also provides a data processing device, including:

[0012] A combining unit, used to combine all the nodes to be combined into at least two tree structures according to a preset combining algorithm;

[0013] A connection unit for connecting the root nodes in at least two tree structures into a ring structure;

[0014] A summarization unit for summarizing the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form the summarized data corresponding to each tree structure;

[0015] A transmission unit for transmitting the summarized data of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure;

[0016] A distribution unit for distributing the summarized data to all child nodes of each tree structure respectively by using the root nodes of each tree structure, so that all child nodes of each tree structure can process the summarized data.

[0017] This application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any one of the above data processing methods when executing the computer program.

[0018] This application also provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program implements the steps of any one of the above data processing methods when executed by a processor.

[0019] This application also provides a computer program product, including a computer program, and the computer program implements the steps of any one of the above data processing methods when executed by a processor.

[0020] Through this application, all nodes to be combined are combined into at least two tree structures, and the root nodes of at least two tree structures are connected into a ring structure. Multiple tree structures process data in parallel. Compared with a single tree structure, it reduces the concentrated transmission of data at a few high-level child nodes. The ring structure connects the root nodes, making the data transmission path more dispersed when the summarized data is transmitted between the root nodes, avoiding the transmission bottleneck of high-level nodes in a single tree structure, thereby improving the overall bandwidth utilization rate of all nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a schematic flowchart of a data processing method provided by an embodiment of this application;

[0023] Figure 2A node framework diagram for data processing provided by an embodiment of the present application;

[0024] Figure 3 A structure diagram of a tree structure provided by an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the data processing process of a tree structure provided by an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the data processing process of a ring structure provided by an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the data processing process of another tree structure provided by an embodiment of the present application;

[0028] Figure 7 A comparison diagram of the node bandwidth utilization provided by an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of a data processing device provided by an embodiment of the present application;

[0030] Figure 9 A schematic diagram of the structure of another data processing device provided by an embodiment of the present application. Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0032] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0033] To enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0034] Describe the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends.

[0035] An embodiment of the present application provides a data processing method. The method will be described in detail in combination with the execution process of the data processing method.

[0036] Figure 1 It is a schematic flowchart of a data processing method provided by an embodiment of the present application.

[0037] As Figure 1 shown, the method includes the following steps:

[0038] Step 101, combine all nodes to be combined into at least two tree structures according to a preset combination algorithm.

[0039] The preset combination algorithm is an algorithm that is preset to combine all nodes to be combined to form at least two tree structures. After obtaining the total number of nodes to be combined, call the preset tree structure layer algorithm and the preset total number of root nodes algorithm to determine the number of tree structure combinations, the number of nodes in each tree structure, and the specific combination method (such as evenly dividing the nodes into node groups and then performing tree structure combination on each node group respectively) to implement constructing all nodes to be combined into a specific tree structure form.

[0040] The nodes to be combined are the basic data units or element nodes that need to be combined to form a tree structure. A tree structure is a hierarchical data structure composed of a root node and several child nodes. Each child node can have its own child nodes, forming a branching structure with clear hierarchical relationships and associations between parent and child nodes.

[0041] The hierarchical relationship of the tree structure helps to optimize the data transmission path, reduce chaos and redundancy during transmission, and improve the efficiency of data transmission.

[0042] Step 102, connect the root nodes in at least two tree structures into a ring structure.

[0043] The ring structure refers to a closed-loop structure formed by connecting the root nodes of multiple tree structures. In a tree structure, the root node is the starting point of the entire tree. It has no parent node and is located at the topmost layer of the tree. All other nodes are derived from the root node. It is the center and foundation of the tree structure, playing a key role in supporting and connecting the entire tree structure.

[0044] First, it is necessary to sort or determine the connection order of the root nodes of all tree structures. This can be achieved in various ways. For example, sorting according to the identifiers of the root nodes, arranging them in ascending or descending order; or determining the connection order according to a certain attribute of the tree structure (such as the size of the tree, the level of the tree, etc.). Suppose we have three tree structures A, B, and C, and their root nodes are Ra, Rb, and Rc respectively. We connect them in the order of Ra, Rb, and Rc. If these root nodes are in the same physical network, they can be connected through network devices (such as switches, routers, etc.). In the actual network topology, configure the network interfaces of the root nodes in a ring connection manner. For example, in an Ethernet environment, connect the network interfaces of Ra, Rb, and Rc to the switch in a ring structure through network cables or optical fibers, so that data can be transmitted along the ring path between these root nodes. A logical ring connection can be achieved by establishing a communication protocol and a data transmission channel. For example, in a distributed system, each root node can run specific communication software or services, and these software or services establish connections through specific communication protocols. In the software, establish communication sessions or connection channels between Ra and Rb, Rb and Rc, and Rc and Ra to form a logical ring structure.

[0045] For a better understanding of the node framework for the entire data processing, as Figure 2 shown, Figure 2 is a node framework diagram for data processing provided by an embodiment of the present application. In Figure 2 , there are 3 tree structures and a ring structure. Among them, each tree structure has 7 nodes, and the ring structure has 4 nodes. However, it should be clear that this statement is not intended to limit the node framework for data processing to only the Figure 2 shown architecture, and it can also be other architectures. The present application does not limit the number of tree structures and ring structures.

[0046] The nodes in the ring structure are arranged in a certain order, and data can be transmitted sequentially along the ring. This enables each root node to only communicate with the two adjacent nodes, simplifies the communication logic, reduces the probability of network congestion and conflicts, and thus ensures that data can flow quickly and efficiently between different tree structures.

[0047] Step 103, summarize the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form the summary data corresponding to each tree structure.

[0048] A child node refers to any node in a tree structure other than the root node. Data refers to the specific information stored and processed in each node, which can be various types of data, such as numerical values, text, images, audio, etc. This data is the object of operation for data processing methods, and through transmission and processing in the tree structure, data sharing, analysis, and utilization are achieved. Aggregated data refers to the data set formed by aggregating and integrating the data of all child nodes in a tree structure. It contains the data information of each child node in the tree structure and is a synthesis and refinement of the child node data.

[0049] There are numerous and scattered child nodes within the tree structure, and aggregated data can integrate the data scattered in each child node to the root node. This is like stringing scattered beads into a single string, facilitating unified management and maintenance of the data.

[0050] Step 104: Transmit the aggregated data of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure.

[0051] When a certain tree structure fails or data is lost, the aggregated data of other tree structures can serve as a backup to assist in restoring and reconstructing the data of the faulty tree structure. This data redundancy mechanism improves the fault tolerance of the system and ensures the continuity and stability of data processing. In addition, by regularly transmitting aggregated data, errors in the data transmission process can be detected and corrected in a timely manner, further enhancing the reliability of the system.

[0052] Step 105: Use the root nodes of each tree structure to distribute the aggregated data to all child nodes of each tree structure so that all child nodes of each tree structure can process the aggregated data.

[0053] The child nodes of different tree structures can simultaneously process the distributed aggregated data, making full use of the computing resources of multiple nodes. The originally centralized data processing tasks are dispersed to each child node, greatly accelerating the data processing speed. For example, in a big data analysis scenario, each child node can independently process part of the data, and overall, the analysis task of the aggregated data can be completed in a shorter time.

[0054] Through this application, all nodes to be combined are combined into at least two tree structures, and the root nodes of at least two tree structures are connected into a ring structure. Multiple tree structures process data in parallel. Compared with a single tree structure, it reduces the centralized transmission of data in a few high-level child nodes. The ring structure connects the root nodes, making the data transmission path more dispersed when the aggregated data is transmitted between the root nodes, avoiding the transmission bottleneck of high-level nodes in a single tree structure, thereby improving the overall bandwidth utilization rate of all nodes.

[0055] As a refinement of step 101, when combining all nodes to be combined into at least two tree structures according to a preset combination algorithm, the following implementation methods can be adopted but are not limited to: obtaining the total sum of the number of all nodes to be combined; calling a preset tree structure layer number algorithm to calculate the total sum to obtain the tree structure layer number; calling a preset root node total number algorithm to calculate the total sum and the tree structure layer number to obtain the total number of root nodes; determining the tree structure combination number and the number of nodes according to the total number of root nodes and the total sum; equally dividing all nodes to be combined into node groups with the number of tree structure combinations; each node group contains the number of nodes to be combined; respectively performing tree structure combination on all nodes to be combined in each node group to obtain at least two tree structures.

[0056] The total sum refers to the total number of all nodes that need to participate in constructing the tree structure. The tree structure combination number is the number of tree structures finally formed by all nodes to be combined. The number of nodes refers to the number of nodes to be combined included in each tree structure. By equally dividing all nodes to be combined into node groups with the combination number, each node group contains the number of nodes to be combined determined above. Any node to be combined in each node group is determined as the root node. The remaining nodes to be combined in the node group except the root node are determined as child nodes. The root node and the child nodes are combined into a tree structure. Dividing the nodes into tree structures facilitates centralized management and resource allocation.

[0057] The calculation formula of the preset tree structure layer number algorithm can be implemented by formula (1):

[0058] L < [log 2 (N + 1)] (1)

[0059] Where L is the tree structure layer number, 2 is the first constant, 1 is the second constant, and N is the total sum.

[0060] The calculation formula of the preset root node total number algorithm can be implemented by formula (2):

[0061]

[0062] Where p is the total number of root nodes, N is the total sum, L is the tree structure layer number, 2 is the first constant, and 1 is the second constant.

[0063] The number of tree structure levels refers to the maximum number of nodes passed from the root node to the farthest leaf node in the tree structure. The number of tree structure levels reflects the depth of the tree structure and is an important indicator for measuring the complexity of the tree structure and the length of the data transfer path. The total number of root nodes refers to the total number of nodes that serve as the starting points of the tree structures in all tree structures. The root node is the core node of the tree structure, responsible for receiving data from child nodes and transferring the data to the root nodes of other tree structures or the ring structure. The combination number refers to the number of combinations of tree structures determined according to the total number of root nodes. Each combination corresponds to a tree structure, and the combination number is equal to the total number of root nodes. The number of nodes refers to the number of nodes included in each tree structure, including the root node and child nodes. The number of nodes is calculated based on the total quantity (the quantity of all nodes to be combined) and the total number of root nodes, indicating the scale of each tree structure.

[0064] By determining the number of tree structure levels and the total number of root nodes, a tree structure with distinct levels and a reasonable structure can be constructed. During the data aggregation and transfer process, data can be transmitted layer by layer within the tree structure, reducing the path length and latency of data transfer. For example, in a tree structure with fewer levels, the data of child nodes can converge to the root node faster, improving the real-time performance of data processing.

[0065] After forming multiple tree structures, each tree structure can process data in parallel. The root nodes of different tree structures can aggregate the data of child nodes simultaneously, and subsequent data transfer and distribution operations can also be carried out in parallel among the tree structures, making full use of system resources and accelerating the overall data processing speed.

[0066] As a refinement of the above embodiment, when performing tree structure combination on all nodes to be combined in each node group respectively to obtain at least two tree structures, any node to be combined in each node group is determined as the root node; the other nodes to be combined in each node group except the root node are determined as child nodes; the root nodes in each node group are respectively combined with the corresponding child nodes to form at least two tree structures. By determining the root node and child nodes, a clear hierarchical relationship is established for each node group. The root node serves as the starting point and center of the tree structure, and the child nodes form branches around the root node. This hierarchical structure helps to clarify the organization and management method of data, making the storage, transmission, and processing of data more orderly and efficient.

[0067] As a refinement of the above embodiment, when performing the determination of the tree structure combination number and the number of nodes according to the total number of root nodes and the total quantity, the following methods can be adopted but are not limited to: determining the total number of root nodes as the tree structure combination number; performing a quotient calculation on the total quantity and the total number of root nodes to obtain the first quotient result; determining the first quotient result as the number of nodes.

[0068] By determining the number of nodes in each tree structure, data processing tasks can be reasonably allocated to each tree structure. Each tree structure undertakes corresponding data processing tasks according to its own number of nodes and processing capabilities, avoiding the situation of overloading a single tree structure. For example, in a distributed computing environment, computing tasks can be evenly distributed to each tree structure according to the number of nodes in each tree structure, improving the overall computing efficiency.

[0069] As a refinement of the above embodiment, when executing the preset total number of root nodes algorithm to calculate the total number and the number of tree structure layers to obtain the total number of root nodes, the following implementation methods can be adopted but are not limited to: obtaining a first constant and a second constant; performing an exponential calculation on the first constant and the number of tree structure layers to obtain a target exponent; performing a quotient calculation on the total number and the target exponent to obtain a second quotient result; performing an addition calculation on the second quotient result and the second constant to obtain the total number of root nodes.

[0070] Specifically, the implementation process of this embodiment is a literal description of formula (2).

[0071] As a refinement of the above embodiment, when executing the preset number of tree structure layers algorithm to calculate the total number to obtain the number of tree structure layers, the following implementation methods can be adopted but are not limited to: performing an addition calculation on the total number and the second constant to obtain a sum result; performing a logarithm calculation on the sum result and the first constant to obtain a target logarithm; determining any value less than the target logarithm as the number of tree structure layers.

[0072] Specifically, the implementation process of this embodiment is a literal description of formula (1).

[0073] As a refinement of step 101, when executing the operation of summarizing the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form the summary data corresponding to each tree structure, the following implementation methods can be adopted but are not limited to: connecting any child node of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure, and connecting each tree structure in the ring structure to any child node of other tree structures in the ring structure; dividing the data of each child node of each tree structure into at least two equal sub-data; transmitting the first equal sub-data among the at least two equal sub-data to the corresponding root node; the first equal sub-data is at least one of the at least two equal sub-data; transmitting the second equal sub-data among the at least two equal sub-data to the root nodes of other tree structures in the ring structure to form the summary data corresponding to each tree structure; the second equal sub-data is the other equal sub-data except the at least one equal sub-data among the at least two equal sub-data.

[0074] Isoquant data is to divide the complete data set of a child node into multiple data blocks of equal size. For example, if a child node has 1,000 units of data, it can be divided into 10 equal quantities of data of 100 units each. By transmitting part of the data (the second equal quantity of data) to the root nodes of other tree structures, the data load can be shared among different tree structures. For example, in a data processing cluster, if the root node of a tree structure is heavily loaded and handles a large number of data aggregation tasks, while the root node of another tree structure is lightly loaded. At this time, transferring part of the child node's data to the root node with a lighter load can allow the root node to share the work of data processing, thereby making the load of the entire system more balanced and improving the bandwidth utilization of each node.

[0075] In practical applications, before using at least two tree nodes to aggregate at least two sub-data to the corresponding root node, different tree structures need to be connected. This can be achieved by, but is not limited to, the following methods: connecting any child node of each tree structure in the ring structure to the root node of other tree structures in the ring structure, and connecting each tree structure in the ring structure to any child node of other tree structures in the ring structure.

[0076] Multiple connection points can disperse the load of data transmission, avoid overloading a single node or path, and achieve more balanced load distribution.

[0077] As a refinement of the above embodiment, when aggregating at least two sub-data to a corresponding root node using at least two tree nodes, it can be implemented in but not limited to the following manner: half of the at least two sub-data are transmitted to each root node of the ring structure; and the other half of the at least two sub-data are transmitted to the root nodes of other tree structures in the ring structure.

[0078] Half of the data refers to the half of the child node data that is equally divided, and this half of the data is transmitted to the target root node. This division can be based on the size, type or other attributes of the data. The other half of the data refers to the remaining part of the child node data except half of the data, and this half of the data is transmitted to the root node of at least one remaining tree structure.

[0079] Parallel transmission and load balancing work together to speed up data aggregation and processing and improve system response time.

[0080] In one possible implementation of the embodiment of the present disclosure, in order to facilitate a better understanding of the tree structure, as shown in FIG. Figure 3 As shown, Figure 3 This is a structural diagram of a tree structure provided by an embodiment of the present application. In the tree structure, 16 is the root node and the remaining nodes are child nodes. In order to facilitate a better understanding of the data processing process of the tree structure, as shown in FIG. Figure 4As shown Figure 4 This is a schematic diagram of the data processing process of a tree structure provided by an embodiment of the present application. The child nodes aggregate their own data to the root node. To facilitate a better understanding of the data processing process of the ring structure, as Figure 5 shown Figure 5 This is a schematic diagram of the data processing process of a ring structure provided by an embodiment of the present application. After the data is aggregated to all the root nodes, the information of all nodes is transmitted to all the root nodes on all the root nodes. To facilitate a better understanding of the data processing process of the tree structure, as Figure 6 shown Figure 6 This is a schematic diagram of another data processing process of a tree structure provided by an embodiment of the present application. After the root node receives all the data, it transmits all the data to all the child nodes. To facilitate a better understanding of the node bandwidth utilization of a single tree structure and the node bandwidth utilization of a combination of multiple tree structures, as Figure 7 shown Figure 7 This is a comparison diagram of the node bandwidth utilization provided by an embodiment of the present application. The node bandwidth of the double binary tree structure is better balanced than that of the single tree structure, and the node bandwidth utilization rate is higher. However, it should be clear that this summary statement is not intended to limit that the combination of multiple tree structures can only be a double binary tree structure, and it can also be a combination of other numbers of tree structures.

[0081] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0082] An embodiment of the present application also provides a data processing device, Figure 8 This is a schematic structural diagram of a data processing device provided by an embodiment of the present application. As Figure 8 shown, it includes:

[0083] Combination unit 21, configured to combine all the nodes to be combined into at least two tree structures according to a preset combination algorithm;

[0084] Connection unit 22, configured to connect the root nodes in at least two tree structures into a ring structure;

[0085] Aggregation unit 23, configured to aggregate the data of all the child nodes of each tree structure in the ring structure to the corresponding root node to form the aggregated data corresponding to each tree structure;

[0086] Transmission unit 24, configured to transmit the aggregated data of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure;

[0087] A distribution unit 25 for distributing the summary data to all child nodes of each tree structure respectively by using the root nodes of each tree structure, so that all child nodes of each tree structure can process the summary data.

[0088] All nodes to be combined are combined into at least two tree structures, and the root nodes of the at least two tree structures are connected into a ring structure. Multiple tree structures process data in parallel. Compared with a single tree structure, it reduces the concentrated transmission of data in a few high-level child nodes. The ring structure connects the root nodes, making the data transmission path more dispersed when the summary data is transmitted between the root nodes, avoiding the transmission bottleneck of high-level nodes in a single tree structure, thereby improving the overall bandwidth utilization rate of all nodes.

[0089] Further, in a possible implementation manner of this embodiment, as Figure 9 shown, the combination unit 21 includes:

[0090] An acquisition module 211 for acquiring the total number of all nodes to be combined;

[0091] A calculation module 212 for calling a preset tree structure layer number algorithm to calculate the total number, and obtaining the number of tree structure layers;

[0092] The calculation module 212 is further configured to call a preset total root node number algorithm to calculate the total number and the number of tree structure layers, and obtain the total number of root nodes;

[0093] A determination module 213 for determining the tree structure combination number and the number of nodes according to the total number of root nodes and the total number;

[0094] A first division module 214 for equally dividing all nodes to be combined into node groups with the number of tree structure combinations; each node group contains the number of nodes to be combined;

[0095] A combination module 215 for respectively performing tree structure combination on all nodes to be combined in each node group to obtain at least two tree structures.

[0096] Further, in a possible implementation manner of this embodiment, the combination module 215 is further configured to

[0097] equally divide all nodes to be combined into node groups with the number of tree structure combinations; each node group contains the number of nodes to be combined;

[0098] respectively perform tree structure combination on all nodes to be combined in each node group to obtain at least two tree structures.

[0099] Further, in a possible implementation manner of this embodiment, the determination module 213 is further configured to

[0100] Determine the total number of root nodes as the number of tree structure combinations;

[0101] Perform a quotient calculation on the sum of quantities and the total number of root nodes to obtain a first quotient result;

[0102] Determine the first quotient result as the number of nodes.

[0103] Furthermore, in a possible implementation manner of this embodiment, the calculation module 212 is further configured to,

[0104] Obtain a first constant and a second constant;

[0105] Perform an exponential calculation on the first constant and the number of tree structure layers to obtain a target exponent;

[0106] Perform a quotient calculation on the sum of quantities and the target exponent to obtain a second quotient result;

[0107] Perform an addition calculation on the second quotient result and the second constant to obtain the total number of root nodes.

[0108] Furthermore, in a possible implementation manner of this embodiment, the calculation module 212 is further configured to,

[0109] Perform an addition calculation on the sum of quantities and the second constant to obtain an addition result;

[0110] Perform a logarithm calculation on the addition result and the first constant to obtain a target logarithm;

[0111] Determine any value less than the target logarithm as the number of tree structure layers.

[0112] Furthermore, in a possible implementation manner of this embodiment, as Figure 9 shown, the summarization unit 23 includes:

[0113] A connection module 231, configured to connect any child node of each tree structure in the ring structure to the root node of other tree structures in the ring structure, and connect each tree structure in the ring structure to any child node of other tree structures in the ring structure;

[0114] A second partitioning module 232, configured to partition the data of each child node of each tree structure into at least two equal sub - data;

[0115] A transmission module 233, configured to transmit the first equal sub - data among at least two equal sub - data to the corresponding root node; the first equal sub - data is at least one of the at least two equal sub - data;

[0116] The transmission module 233 is further configured to transmit the second equal - quantum data among at least two equal - quantum data to the root nodes of other tree structures in the ring structure, so as to form summary data corresponding to each tree structure; the second equal - quantum data is the other equal - quantum data except at least one equal - quantum data among the at least two equal - quantum data.

[0117] For the description of the features in the corresponding embodiments of the data processing device, reference can be made to the relevant descriptions in the corresponding embodiments of the data processing method, which will not be elaborated here one by one.

[0118] An embodiment of the present application further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above - mentioned embodiments of the data processing method.

[0119] An embodiment of the present application further provides a computer - readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above - mentioned embodiments of the data processing method when running.

[0120] In an exemplary embodiment, the above - mentioned computer - readable storage medium may include, but is not limited to: USB flash drives, read - only memories (ROM), random access memories (RAM), mobile hard disks, magnetic disks, or optical discs and other media that can store computer programs.

[0121] An embodiment of the present application further provides a computer program product. The above - mentioned computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above - mentioned embodiments of the data processing method.

[0122] An embodiment of the present application further provides another computer program product, including a non - volatile computer - readable storage medium. The non - volatile computer - readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above - mentioned embodiments of the data processing method.

[0123] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0124] The above has introduced in detail a method and apparatus for processing data, an electronic device, and a storage medium provided in this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that: include: According to the preset combination algorithm, all nodes to be combined are combined into at least two tree structures; Connecting the root nodes in the at least two tree structures into a ring structure; Aggregate the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form the summary data corresponding to each tree structure; Transferring the summary data of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure; The summary data is distributed to all child nodes of each tree structure using the root node of each tree structure, so that all child nodes of each tree structure process the summary data.

2. The data processing method according to claim 1, characterized in that: Combining all the nodes to be combined into at least two tree structures according to the preset combination algorithm includes: Obtaining the total number of all nodes to be combined; Calling a preset tree structure layer number algorithm, calculating the sum of the quantities, and obtaining the tree structure layer number; Calling a preset root node total number algorithm, calculating the sum of the number and the number of tree structure layers, to obtain the total number of root nodes; Determine the number of tree structure combinations and the number of nodes according to the total number of root nodes and the sum of the number; All the nodes to be combined are equally divided into node groups of the number of tree structure combinations; each node group contains the number of nodes to be combined; All the nodes to be combined in each node group are respectively combined into a tree structure to obtain the at least two tree structures.

3. The data processing method according to claim 2, characterized in that: The step of performing tree structure combination on all nodes to be combined in each node group to obtain the at least two tree structures comprises: Determine any node to be combined in each node group as a root node; Determine other nodes to be combined in each node group except the root node as child nodes; The root node in each node group is combined with the corresponding child node in a tree structure to obtain the at least two tree structures.

4. The data processing method according to claim 2, characterized in that: Determining the number of tree structure combinations and the number of nodes according to the total number of root nodes and the sum of the number includes: Determine the total number of root nodes as the number of tree structure combinations; Calculate the quotient of the total number of the number and the total number of the root nodes to obtain a first quotient value result; The first quotient result is determined as the number of nodes.

5. The data processing method according to claim 2, characterized in that: The calling of the preset root node total number algorithm to calculate the sum of the number and the number of tree structure layers to obtain the total number of root nodes includes: Get the first constant and the second constant; Performing exponential calculation on the first constant and the number of tree structure layers to obtain a target index; Calculate the quotient of the sum of the quantities and the target index to obtain a second quotient value result; The second quotient result and the second constant are added together to obtain the total number of root nodes.

6. The data processing method according to claim 5, characterized in that: The calling of the preset tree structure layer number algorithm to calculate the sum of the quantities to obtain the tree structure layer number includes: Adding the sum of the quantities and the second constant to obtain a sum result; Performing logarithm calculation on the sum result and the first constant to obtain a target logarithm; Any value smaller than the target number of logarithms is determined as the number of tree structure layers.

7. The data processing method according to claim 1, characterized in that: Aggregating the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form the aggregated data corresponding to each tree structure includes: Connecting any child node of each tree structure in the ring structure to the root node of the other tree structures in the ring structure, and connecting each tree structure in the ring structure to any child node of the other tree structures in the ring structure; Dividing the data of each child node of each tree structure into at least two equal sub-data; Transmitting a first equal sub-quantum data of the at least two equal sub-quantum data to a corresponding root node; the first equal sub-quantum data is at least one equal sub-quantum data of the at least two equal sub-quantum data; The second isoquantum data of the at least two isoquantum data is transmitted to the root node of the other tree structures in the ring structure to form summary data corresponding to each tree structure; the second isoquantum data is other isoquantum data of the at least two isoquantum data except the at least one isoquantum data.

8. A data processing device, characterized in that: include: A combining unit, used to combine all the nodes to be combined into at least two tree structures according to a preset combining algorithm; A connecting unit, used to connect the root nodes in the at least two tree structures into a ring structure; A summarizing unit, used for summarizing the data of all child nodes of each tree structure in the ring structure to the corresponding root node to form summary data corresponding to each tree structure; A transmission unit, used for transmitting the summary data of each tree structure in the ring structure to the root nodes of other tree structures in the ring structure; The distribution unit is used to distribute the summary data to all the child nodes of each tree structure using the root node of each tree structure, so that all the child nodes of each tree structure can process the summary data.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 7.

Images