Data distribution method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a data distribution method and device, electronic equipment and a storage medium. The method comprises the following steps of: receiving to-be-processed data from an upstream system at each computing node; determining whether the current computing node is in a receivable data state or not according to the state of the data queue corresponding to the current computing node; if the current computing node is in a data receivable state, writing the to-be-processed data into a data queue corresponding to the current computing node, so that the current computing node reads the to-be-processed data from the corresponding data queue for processing; and otherwise, forwarding the to-be-processed data to other computing nodes. In this way, the data flow direction of the to-be-processed data among the computing nodes is adjusted according to the actual processing capacity of the current computing node, load balancing among the computing nodes is achieved at the computing nodes, on one hand, computing resources are more fully utilized, and the resource utilization rate is increased; and on the other hand, the overall data processing efficiency is improved on the whole.

Description

Technical Field

[0001] The present application relates to the field of big data and artificial intelligence technology, and in particular to a data distribution method, device, electronic device and storage medium. Background Art

[0002] In order to improve data processing efficiency, distributed systems have emerged and are applied to various fields and scenarios. For example, with the rapid development of big data and artificial intelligence technologies, training machine learning models based on large-scale data sets has become a common and important business requirement. In order to accelerate model training, training tasks are distributed to multiple computing nodes for parallel processing.

[0003] However, in practical applications, there are differences in the data allocation and processing capabilities of each computing node. This difference may cause some computing nodes to be overloaded while others are idle. On the one hand, computing resources are not fully utilized, resulting in resource waste, and on the other hand, the overall data processing efficiency is affected. Summary of the invention

[0004] The present application provides a data allocation method, device, electronic device and storage medium to more fully utilize computing resources and improve overall data processing efficiency.

[0005] This application provides the following solutions:

[0006] In a first aspect, a data distribution method is provided, which is applied to a distributed system including a plurality of computing nodes, wherein to-be-processed data from an upstream system is distributed to the plurality of computing nodes, and each computing node is respectively executed as a current computing node:

[0007] Receive data to be processed from upstream systems;

[0008] Determining whether the current computing node is in a state capable of receiving data according to the state of the data queue corresponding to the current computing node;

[0009] When the current computing node is in a state where it can receive data, the data to be processed is written into a data queue corresponding to the current computing node, so that the current computing node reads the data to be processed from the corresponding data queue for processing;

[0010] When the current computing node is in a state where it cannot receive data, the data to be processed is forwarded to other computing nodes based on a list of available nodes recorded by the current node, wherein the list of available nodes includes computing nodes in the distributed system that are in a state where they can receive data except the current computing node.

[0011] Optionally, determining whether the current computing node is in a state capable of receiving data according to a state of a data queue corresponding to the current computing node includes:

[0012] In response to a state indicator value of a data queue corresponding to the current computing node being less than a first upper threshold, switching the current computing node to a state capable of receiving data, wherein the state indicator value is used to characterize the degree to which the data queue is occupied;

[0013] In response to the state indicator value of the data queue corresponding to the current computing node being greater than a second upper limit threshold, switching the current computing node to a state where data cannot be received;

[0014] The first upper threshold is less than or equal to the second upper threshold.

[0015] Optionally, the method further comprises:

[0016] When the actual resource occupancy status of the current computing node is lower than a first preset status for a certain period of time, increasing the first upper limit threshold and / or the second upper limit threshold;

[0017] When the actual resource occupancy status of the current computing node is higher than a second preset status for a certain period of time, reducing the first upper limit threshold and / or the second upper limit threshold, and the first preset status is lower than or equal to the second preset status;

[0018] The resource occupancy status includes at least one of a processor occupancy status, a memory occupancy status and a network bandwidth occupancy status.

[0019] Optionally, the method further comprises:

[0020] The current computing node sends information about whether the current computing node is in a state capable of receiving data to other computing nodes in the distributed system;

[0021] The available node list is updated by using the received information on whether the data can be received and sent by other computing nodes.

[0022] Optionally, the updating of the available node list by using the received information on whether the data can be received and sent by other computing nodes comprises: recording other computing nodes in a state of being able to receive data in the available node list by using the information on whether the data can be received and sent by other computing nodes, and updating the order of each computing node in the available node list by using the communication delay between other computing nodes;

[0023] Forwarding the data to be processed to other computing nodes according to the available node list recorded by the current node includes: distributing the data to be processed to one or more computing nodes in the available node list according to the sorting results of each computing node in the available node list recorded by the current computing node.

[0024] Optionally, forwarding the to-be-processed data to other computing nodes includes:

[0025] Convert the data to be processed into a serialized format to obtain a byte stream;

[0026] The byte stream is transmitted to other computing nodes using remote procedure call (RPC).

[0027] Optionally, the method further comprises:

[0028] The current computing node receives data to be processed from other computing nodes in the distributed system;

[0029] The received data to be processed is written into the data queue corresponding to the current computing node.

[0030] In a second aspect, a data distribution device is provided, which is applied to a distributed system including a plurality of computing nodes, to which to-be-processed data from an upstream system are distributed, and the device is respectively arranged at each computing node; the device comprises:

[0031] A forwarding processing unit, configured to receive data to be processed from an upstream system;

[0032] A state determination unit, configured to determine whether the current computing node is in a state capable of receiving data according to a state of a data queue corresponding to the current computing node;

[0033] The forwarding processing unit is further configured to, when the current computing node is in a state where it can receive data, write the data to be processed into a data queue corresponding to the current computing node, so that the current computing node can read the data to be processed from the corresponding data queue for processing; when the current computing node is in a state where it cannot receive data, forward the data to be processed to other computing nodes based on a list of available nodes recorded by the current node, wherein the list of available nodes includes computing nodes in the distributed system that are in a state where they can receive data except the current computing node.

[0034] Optionally, the state determination unit is specifically configured as follows: in response to a state indicator value of the data queue corresponding to the current computing node being less than a first upper limit threshold, switching the current computing node to a state where data can be received, the state indicator value being used to characterize the degree to which the data queue is occupied; in response to a state indicator value of the data queue corresponding to the current computing node being greater than a second upper limit threshold, switching the current computing node to a state where data cannot be received; wherein the first upper limit threshold is less than or equal to the second upper limit threshold.

[0035] Optionally, the state determination unit may also be configured to: when the actual resource occupancy status of the current computing node is lower than a first preset status for a certain period of time, increase the first upper limit threshold and / or the second upper limit threshold;

[0036] When the actual resource occupancy status of the current computing node is higher than a second preset status for a certain period of time, reducing the first upper limit threshold and / or the second upper limit threshold, and the first preset status is lower than or equal to the second preset status;

[0037] The resource occupancy status includes at least one of a processor occupancy status, a memory occupancy status and a network bandwidth occupancy status.

[0038] Optionally, the forwarding processing unit may also be configured to: send information on whether the current computing node is in a state capable of receiving data to other computing nodes in the distributed system.

[0039] The state determination unit may also be configured to update the available node list by using the information on whether the data can be received and sent by other computing nodes.

[0040] Optionally, the state determination unit may be specifically configured to: record other computing nodes in a state of being able to receive data in the available node list by using the information on whether the other computing nodes are in a state of being able to receive data sent by the other computing nodes, and update the order of each computing node in the available node list by using the communication delay between the other computing nodes;

[0041] The forwarding processing unit is specifically configured to distribute the to-be-processed data to one or more computing nodes in the available node list according to the sorting result of each computing node in the available node list recorded by the current computing node.

[0042] Optionally, the forwarding processing unit is specifically configured to: convert the data to be processed into a serialized format to obtain a byte stream; and transmit the byte stream to other computing nodes using a remote procedure call RPC.

[0043] Optionally, the forwarding processing unit is specifically configured to: receive data to be processed from other computing nodes in the distributed system; and write the received data to be processed into a data queue corresponding to the current computing node.

[0044] Optionally, the forwarding processing unit is specifically configured to: receive byte streams from other computing nodes in the distributed system; and process the byte streams using streaming RPC to obtain data to be processed.

[0045] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the first aspect are implemented.

[0046] In a fourth aspect, an electronic device is provided, including:

[0047] one or more processors; and

[0048] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0049] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0050] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0051] The present application performs status judgment, load balancing control and status recording of other computing nodes at each computing node, that is, determines whether the current computing node is in a state where it can receive data based on the state of the data queue corresponding to the current computing node, and writes the data to be processed into the data queue corresponding to the current computing node when it is in a state where it can receive data; otherwise, forwards the data to be processed to other computing nodes based on the list of available nodes recorded by the current node. In this way, the data flow of the data to be processed between the computing nodes is adjusted according to the actual processing capacity of the current computing node, and load balancing between the computing nodes is achieved at the computing node. On the one hand, computing resources are more fully utilized and resource utilization is improved; on the other hand, the overall data processing efficiency is improved as a whole; on the other hand, other computing nodes that can receive data in the distributed system can be determined efficiently and timely, further improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a system architecture diagram applicable to the embodiments of the present application;

[0054] Figure 2 A flow chart of a data allocation method provided in an embodiment of the present application;

[0055] Figure 3 A schematic block diagram of a data distribution device provided in an embodiment of the present application;

[0056] Figure 4 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0058] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0059] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0060] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0061] At present, there are some technologies that use distributed systems to schedule data from upstream systems and share it among multiple computing nodes for data processing, thereby achieving load balancing. However, the current scheduling method usually distributes the data of the upstream system to multiple computing nodes using a preset scheduling strategy. For example, the production data is divided into different batches, and different batches of data are shared among different computing nodes. However, due to the differences in the actual operating conditions and processing capabilities of each computing node, this scheduling method will still result in some computing nodes being overloaded while others are idle, resulting in a waste of resources and affecting the overall training efficiency.

[0062] In view of this, the present application provides a new idea. In order to facilitate the understanding of the present application, the system architecture on which the present application is based is first described. Figure 1 An exemplary system architecture to which the embodiments of the present application can be applied is shown. Figure 1 As shown in , a distributed system is used for data processing under the system architecture, and the distributed system includes multiple computing nodes. The distributed system connects multiple computing nodes together through a network, and each computing node independently performs a data processing task, and the data processing task includes corresponding computing and storage tasks.

[0063] The upstream system usually corresponds to the data production system, which is used to send the produced data as the data to be processed to the distributed system in batches. The upstream system can use the produced data to generate batch data, and send the batch data randomly or according to a preset strategy to the computing nodes in the distributed system. For example, after a batch of data is generated, a random number is randomly generated, and according to the correspondence between the preset random number and the computing node, the batch of data is sent to the computing node corresponding to the random number.

[0064] The upstream system may adopt a Kafka system or a system using other mechanisms.

[0065] In the embodiment of the present application, a corresponding current queue and data distribution device are set in each computing node. The current queue only stores the data to be processed of the current computing node, that is, the data processing device in the computing node only reads the data to be processed from the current queue corresponding to the current computing node for data processing.

[0066] The data distribution device is used to further distribute the data to be processed received by the current computing node from the upstream system using the method provided in the embodiment of the present application, including distribution to the current computing node and distribution to other computing nodes. The specific data distribution method can be found in the detailed description of the subsequent embodiments.

[0067] The above-mentioned computing nodes can be set in a single server, a server group consisting of multiple servers, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in a cloud computing service system to solve the defects of difficult management and weak service scalability in traditional physical hosts and virtual private servers (VPS) services. Alternatively, the above-mentioned computing nodes can also be set in a computer terminal with high processing power, and this application does not impose any special restrictions on this.

[0068] It should be understood that Figure 1 The number of upstream systems and computing nodes in the example is only for illustration purposes. Any number of upstream systems and computing nodes may be provided as required.

[0069] Figure 2 A flow chart of a data distribution method provided in an embodiment of the present application, which can be Figure 1 The distributed system shown in FIG. 1 is executed by each computing node, and specifically can be executed by the data distribution device of each computing node. Figure 2 As shown in , the method may include the following steps:

[0070] Step 201: Receive data to be processed from an upstream system.

[0071] Step 203: Determine whether the current computing node is in a state where it can receive data based on the state of the data queue corresponding to the current computing node. If the current computing node is in a state where it can receive data, execute step 205; if the current computing node is in a state where it cannot receive data, execute step 207.

[0072] Step 205: Write the data to be processed into the data queue corresponding to the current computing node, so that the current computing node can read the data to be processed from the corresponding data queue for processing.

[0073] Step 207: forward the data to be processed to other computing nodes according to the available node list recorded by the current node.

[0074] It can be seen from the above process that after receiving the data to be processed, the computing node in this application adjusts the data flow of the data to be processed between the computing nodes according to the actual processing capacity of the computing node, thereby realizing load balancing between the computing nodes. On the one hand, it makes more effective use of computing resources and improves resource utilization; on the other hand, it improves the overall data processing efficiency as a whole; on the other hand, it can efficiently and timely determine other computing nodes in the distributed system that can receive data, thereby further improving data processing efficiency.

[0075] In essence, it is equivalent to adding a global shuffle mechanism to the computing nodes. Figure 1 The data distribution device arranged in the computing node can be regarded as a shuffle unit.

[0076] The key steps in the above process and the effects that can be further produced are described in detail below in conjunction with the embodiments. It should be noted that the limitations such as "first" and "second" involved in the present disclosure do not have limitations in terms of size, order, and quantity, and are only used to distinguish them in name. For example, "first upper limit threshold" and "second upper limit threshold" are used to distinguish two thresholds.

[0077] The above step 203, namely "determining whether the current computing node is in a state capable of receiving data according to the state of the data queue corresponding to the current computing node", is described in detail below in conjunction with an embodiment.

[0078] In an embodiment of the present application, each computing node is provided with a corresponding data queue, which can be a local buffer space of the computing node, used to temporarily store the data to be processed assigned to the current computing node, so that the data processing unit of the current computing node can read the data to be processed from the data queue and perform data processing.

[0079] In the embodiment of the present application, after the Shuffle Unit receives the data to be processed from the upstream, it first determines the state of the current computing node, which may include a state of receiving data and a state of not receiving data. The state of the current computing node can be determined according to the state indicator value of the data queue corresponding to the current computing node, and the state indicator value is used to characterize the degree of occupancy of the data queue.

[0080] For example, in response to the state indicator value of the data queue corresponding to the current computing node being less than the first upper threshold value (lower_bound), the current computing node is switched to a state where it can receive data; in response to the state indicator value of the data queue corresponding to the current computing node being greater than the second upper threshold value (upper_bound), the current computing node is switched to a state where it cannot receive data.

[0081] The first upper limit threshold may be equal to the second upper limit threshold. However, in order to avoid frequent switching of the state of the computing node, it is preferred to set the first upper limit threshold to be smaller than the second upper limit threshold.

[0082] For example, for the current computing node, initially the state indicator value of the data queue corresponding to the current computing node is less than lower_bound, so it is in a state where data can be received. When the state indicator value of the data queue corresponding to the computing node is between lower_bound and upper_bound, the state of the current computing node is kept unchanged until the state indicator value of the data queue corresponding to the current computing node is greater than the second upper threshold value upper_bound, and the state of the current computing node is switched to a state where data cannot be received. If the state indicator value of the corresponding data queue of the current computing node decreases after the processing of the data to be processed by the current computing node, it needs to be reduced until the state indicator value of the data queue corresponding to the current computing node is less than lower_bound before the state of the current computing node is switched to a state where data can be received. However, if the first upper threshold value and the second upper threshold value are equal (i.e., they correspond to one threshold value in this case), it may happen that the state indicator value of the data queue corresponding to the computing node fluctuates near the threshold value (i.e., the state indicator value frequently changes above and below the threshold value in a short period of time), which will cause the state of the computing node to be switched continuously. This disadvantage can be effectively avoided by retaining a certain interval between the first upper threshold value and the second upper threshold value.

[0083] The status indicator value of the above-mentioned data queue can be reflected by the number of pending data in the data queue. For example, if the number of pending data in the data queue is less than lower_bound, it is determined that the current computing node is in a state where it can receive data; if the number of pending data in the data queue is greater than upper_bound, it is determined that the current computing node is in a state where it cannot receive data.

[0084] In addition, the status indicator value of the above-mentioned data queue can also be reflected through other means, for example, through the proportion of the data queue occupied by the data to be processed. If the proportion of the data queue occupied by the data to be processed is less than lower_bound, it is determined that the current computing node is in a state where it can receive data; if the proportion of the data queue occupied by the data to be processed is greater than upper_bound, it is determined that the current computing node is in a state where it cannot receive data.

[0085] If the current computing node is in a state where it can receive data, it means that the data processing unit in the current computing node still has sufficient data processing capacity to process new data to be processed, so step 205 can be executed to write the data to be processed from the upstream system into the data queue corresponding to the current computing node.

[0086] If the current computing node is in a state where it can receive data, it means that the data processing unit in the current computing node does not have enough data processing capacity to process the new data to be processed. Therefore, step 207 can be executed to forward the data to be processed to other computing nodes. Step 207 is described in detail below in conjunction with the embodiments.

[0087] In the embodiment of the present application, when forwarding the data to be processed to other computing nodes, a computing node can be randomly selected for forwarding. However, as a more preferred implementation, the data to be processed can be distributed to one or more computing nodes in the available node list based on the available node list maintained by the current computing node.

[0088] In this implementation, each computing node may maintain an available node list, where the available node list includes computing nodes in the distributed system that are in a state of being able to receive data except the current computing node.

[0089] Each computing node may notify each other whether the current computing node is in a state where it can receive data. As one achievable method, each computing node may periodically notify other computing nodes whether the current computing node is in a state where it can receive data.

[0090] As another achievable manner, each computing node can notify other computing nodes whether the current computing node is in a state where data can be received when the state of the current computing node changes. For example, if the current computing node changes from a state where data can be received to a state where data cannot be received, the information that the current computing node is in a state where data cannot be received is notified to other computing nodes. If the current computing node changes from a state where data cannot be received to a state where data can be received, the information that the current computing node is in a state where data can be received is notified to other computing nodes. After receiving notifications from other computing nodes, the computing node maintains an available node list (i.e., adds or deletes computing nodes in the available node list) according to the received notifications, so that the available node list always contains information about computing nodes that are in a state where data can be received.

[0091] When forwarding the data to be sent, the computing node may randomly select a computing node from the available node list as one of the achievable ways, and forward the data to be sent to the randomly selected computing node.

[0092] As another achievable manner, N computing nodes are randomly selected from the available node list, where N is a positive integer greater than 1, and the data to be sent is divided into N parts and then distributed to the N computing nodes.

[0093] As another feasible method, when receiving notifications from other computing nodes, the computing nodes in the available node list can be sorted according to the actual communication delay. The computing nodes with smaller communication delays are ranked higher. The computing nodes can preferentially select one or more computing nodes ranked higher to distribute the data to be processed according to the sorting results. Among them, the computing nodes in the available node list can be updated in real time according to the actual communication delay status.

[0094] For example, assume that the distributed system includes computing nodes A, B, C, D, and E. After computing node A receives notifications from computing nodes B, C, D, and E, it determines that the computing nodes that can receive data are computing nodes C, D, and E. Based on the received notifications, computing node A determines that the transmission delays of computing nodes C, D, and E are 20ms, 30ms, and 50ms, respectively. Then the available node list maintained by computing node A can be:

[0095] Compute Node C>-Compute Node D>-Compute Node E

[0096] Among them, “>-” represents the order of sorting.

[0097] After receiving the data to be processed, computing node A determines that the current computing node A is in a state where it cannot receive data. Then, computing node C is preferentially selected from the available node list according to the order of each computing node, and the data to be processed is forwarded to computing node C.

[0098] This implementation method is equivalent to providing a way to intelligently select computing nodes for data distribution, giving priority to computing nodes with the smallest communication delay for distributing data to be processed, thereby minimizing data transmission delays and improving data transmission efficiency and performance.

[0099] It should be noted that in addition to sorting the computing nodes in the available node list according to the communication delay, other sorting strategies can also be used, such as sorting based on the occupancy status of the data queues corresponding to the computing nodes, and prioritizing the data queues with low occupancy rates.

[0100] In an embodiment of the present application, when data is transmitted between computing nodes, a streaming data transmission method can be adopted. When the computing node transmits the data to be sent to other computing nodes, the data to be processed can be serialized and converted to obtain a byte stream, for example, it can be converted by a serialization format such as ProtoBuf (Protocal Buffer, a data structure serialization framework), so as to facilitate network transmission. Then use RPC (Remote Procedure Call) to transmit the byte stream to other computing nodes, for example, multiple RPC frameworks such as bRPC, Apache Thrift, gPRC, etc. can be used to ensure that the serialized byte stream can be sent to other computing nodes through a network transmission protocol.

[0101] For computing nodes, in addition to receiving data to be processed from upstream systems, they may also receive data to be processed from other computing nodes. As mentioned above, the data to be processed from other computing nodes is usually transmitted in the form of byte streams. When a computing node receives a byte stream sent by other computing nodes, it can use a streaming data processing framework to process the received data stream, including data analysis (for example, verifying the integrity, format correctness, consistency, etc. of the data), data conversion (the process of deserialization, converting the byte stream into the original format of the data to be processed) and data aggregation (the data to be processed that is usually transmitted is packet data, so it is necessary to perform data aggregation on the packet data) and other processing to obtain the data to be processed. Among them, the streaming data processing framework can adopt such as streaming RPC.

[0102] Through streaming technologies such as "bRPC+streaming RPC", efficient and reliable streaming data transmission can be achieved, improving the efficiency of global shuffle and the performance of distributed systems. As a high-performance communication framework with a highly reliable fault-tolerant mechanism, bRPC supports data transmission from multiple computing nodes and confirms it at the receiving end to ensure data reliability, and its connection pool and connection reuse functions can further enhance the stability and efficiency of communication. Streaming RPC allows data to be received and processed uninterruptedly during transmission, without waiting for all data to arrive before processing, thereby improving the performance and efficiency of distributed systems.

[0103] As one of the feasible ways, the computing node can directly write the pending data from other computing nodes into the data list corresponding to the current computing node without having to make judgments based on the status, thereby avoiding multiple transmissions of the pending data between computing nodes. On the one hand, it saves network bandwidth, and on the other hand, it reduces the impact on the performance of the distributed system.

[0104] In order to prevent the situation where the list of available nodes maintained by the computing node is not updated in time and the status changes of some available nodes are not recorded in time in some cases, as another feasible way, the computing node can also first determine whether the current computing node is in a state where it can receive data for the pending data from other computing nodes. If so, the pending data is written into the data queue corresponding to the current computing node; otherwise, the pending data will continue to be forwarded to other computing nodes.

[0105] Furthermore, as described in the previous embodiment, in the embodiment of the present application, whether the computing node is in a state where data can be received is determined based on the relationship between the state of the data queue and the first upper threshold (lower_bound) and the second upper threshold (upper_bound). As one of the achievable methods, the first upper threshold and / or the second upper threshold can be preset, for example, using an empirical value or an experimental value.

[0106] As another achievable manner, the first upper threshold and / or the second upper threshold may be adjusted according to the actual resource occupancy status of the computing node, wherein the resource occupancy status may include but is not limited to at least one of processor occupancy status, memory occupancy status and network bandwidth occupancy status.

[0107] The processor occupancy status may be the occupancy rate of the data allocation unit (Shuffle Unit) of the current computing node to the CPU, or the occupancy rate of the data processing unit to a GPU, etc. The memory occupancy status may be the occupancy rate of the memory of the current computing node. The network bandwidth occupancy status may be the occupancy rate of the network bandwidth of the current computing node.

[0108] For example, when the processor occupancy of the current computing node is at a low level (lower than the first preset state) for a certain period of time, it means that the current computing node lacks data to be processed, and the current processing capacity is strong, and the first upper limit threshold and the second upper limit threshold can be increased. Among them, increasing the first upper limit threshold can make the current computing node more likely to be in a state where data can be received, thereby supplementing the data to be processed. Increasing the second upper limit threshold can enable the current computing node to switch to a state where data cannot be received after receiving more data to be processed. When the processing occupancy of the current computing node is at a high level (higher than the second preset state) for a certain period of time, it means that there is too much data to be processed in the current computing node, and the current processing capacity is weak, and the first upper limit threshold and the second upper limit threshold can be reduced. Among them, reducing the first upper limit threshold can make it more difficult for the current computing node to be in a state where data can be received, so that new data to be processed is not received, and the data to be processed that the current computing node has already received is processed. Reducing the second upper limit threshold can make it easier for the current computing node to be in a state where data cannot be received, thereby processing the data to be processed that has already been received. Among them, the above-mentioned first preset state is lower than or equal to the second preset state.

[0109] Obviously, the method of dynamically adjusting the first upper threshold and / or the second upper threshold in the embodiment of the present application can better adapt to changes in the environment and workload.

[0110] The above method provided in the embodiment of the present application can be applied to a variety of application scenarios, including but not limited to:

[0111] Application scenario 1: model training scenario

[0112] With the rapid development of big data and artificial intelligence technologies, machine learning model training based on large-scale data sets has become a common and important business requirement. Therefore, distributed systems are often used to complete machine learning model training tasks. Usually, training tasks are assigned to multiple computing nodes for parallel computing to accelerate the model training process.

[0113] In this application scenario, after the upstream system generates batch data from the training data set, the batch data is sent to the computing nodes in the distributed system randomly or according to a certain strategy. The above batch data refers to a batch of training data. Each computing node in the distributed system can adopt the method provided by the embodiment of the present application. After the Shuffle Unit receives the training data from the upstream system, it determines whether the current computing node is in a state where it can receive data according to the state of the data queue corresponding to the current computing node; if the current computing node is in a state where it can receive data, the received training data is written to the data queue corresponding to the current computing node, so that the GPU of the current computing node can read the training data from the corresponding data queue for model training processing; otherwise, the training data is forwarded to other computing nodes.

[0114] Usually in this application scenario, GPU is preferred as the data processing device in the computing node used for model training due to its powerful parallel computing capabilities, floating-point computing capabilities, support for deep learning frameworks, high memory bandwidth, etc. Shuffle Unit can be implemented by running a process on the CPU, and the data queue can use cache space.

[0115] Application scenario 2: Video distribution scenario

[0116] As video services continue to develop, the number of users and the amount of video content are increasing. Distributed systems are widely used in video distribution scenarios due to their good scalability. In video distribution scenarios, the popularity of different video content may vary greatly, resulting in some computing nodes being overloaded while others are underloaded.

[0117] Therefore, in this application scenario, the upstream system can send the video data produced by the content producer to the computing nodes in the distributed system randomly or using a certain strategy. Each computing node in the distributed system can adopt the method provided by the embodiment of the present application. After the Shuffle Unit receives the video data from the upstream system, it determines whether the current computing node is in a state where it can receive data according to the state of the data queue corresponding to the current computing node; if the current computing node is in a state where it can receive data, the received video data is written to the data queue corresponding to the current computing node, so that the current computing node can read the video data from the corresponding data queue for video processing (including video transcoding, content review, cover image generation, video storage, etc.); otherwise, the video data is forwarded to other computing nodes.

[0118] In this application scenario, the data processing device in the computing node can be implemented by the CPU on the server side. If the video processing has a high real-time requirement, it can also be implemented by the GPU on the server side. The Shuffle Unit can be implemented by a process running on the CPU on the server side, and the data queue can use the cache space on the server side.

[0119] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0120] In order to verify the effect of the method provided in the embodiment of the present application, the method provided in the embodiment of the present application was experimented. In order to make the experimental data more comparable, under the same experimental conditions, the experimental results of the method provided in the embodiment of the present application (hereinafter referred to as global shuffling), no shuffling and basic shuffling were compared.

[0121] The No Shuffle method means that each computing node in the distributed system only processes the data to be processed that it receives, and there is no data transmission between computing nodes.

[0122] The Basic Shuffle method means that after each computing node in the distributed system receives the data to be processed, it evenly distributes the data to be processed to other computing nodes without judging its own status.

[0123] The upstream system sends training data sets of the same size to the computing nodes in the distributed system in batches for model training. The sample accumulation conditions obtained by the above three methods are compared. In the non-shuffle method, the lag time corresponding to each partition data is very large. In the basic shuffle method, the lag corresponding to each partition data is significantly reduced. In the global shuffle method, the lag of each partition data is further reduced. Therefore, global shuffle performs best in reducing the lag of training data samples, which means that the data processing efficiency of the distributed system is optimal under the global shuffle method.

[0124] The network overhead, CPU usage and model AUC (Area Under Curve) corresponding to the three methods are compared. AUC mainly reflects the performance of the model, which is usually used to indicate the reasoning ability of the model and is one of the commonly used indicators for evaluating model performance in machine learning.

[0125] No shuffle has the lowest network overhead because there is no data exchange between computing nodes. Global shuffle also has a smaller network overhead than basic shuffle. Although the CPU usage of global shuffle is higher than that of no shuffle, it is lower than that of basic shuffle. Global shuffle has the best model AUC performance among the three methods, indicating that the model training effect is the best. Taking various factors into consideration, global shuffle is the method with the best overall effect.

[0126] Figure 3 A schematic block diagram of a data distribution device provided in an embodiment of the present application, wherein the device is arranged at Figure 1 The computing nodes in the architecture shown. Figure 3 As shown in , the device 300 may include: a forwarding processing unit 301 and a state determination unit 302. The main functions of each component unit are as follows:

[0127] The forwarding processing unit 301 is configured to receive data to be processed from an upstream system.

[0128] The state determination unit 302 is configured to determine whether the current computing node is in a state capable of receiving data according to the state of the data queue corresponding to the current computing node.

[0129] The forwarding processing unit 301 can be further configured to write the data to be processed into the data queue corresponding to the current computing node when the current computing node is in a state where it can receive data, so that the current computing node can read the data to be processed from the corresponding data queue for processing; when the current computing node is in a state where it cannot receive data, forward the data to be processed to other computing nodes based on the list of available nodes recorded by the current node, and the list of available nodes includes computing nodes in the distributed system that are in a state where they can receive data except the current computing node.

[0130] As one of the implementable ways, the state determination unit 302 can be specifically configured as follows: in response to the state indicator value of the data queue corresponding to the current computing node being less than the first upper threshold, the current computing node is switched to a state where data can be received; in response to the state indicator value of the data queue corresponding to the current computing node being greater than the second upper threshold, the current computing node is switched to a state where data cannot be received; wherein the first upper threshold is less than or equal to the second upper threshold. The state determination unit 302 can be regarded as a switch whether to receive data.

[0131] Furthermore, the status determination unit 302 can also be configured as: when the actual resource occupancy status of the current computing node is lower than the first preset status for a certain period of time, increasing the first upper limit threshold and / or the second upper limit threshold; when the actual resource occupancy status of the current computing node is higher than the second preset status for a certain period of time, reducing the first upper limit threshold and / or the second upper limit threshold, and the first preset status is lower than or equal to the second preset status; wherein the resource occupancy status includes at least one of the processor occupancy status, the memory occupancy status and the network bandwidth occupancy status.

[0132] Furthermore, the forwarding processing unit 301 may also be configured to: send information on whether the current computing node is in a state where data can be received to other computing nodes in the distributed system. Accordingly, the state determination unit 302 may also be configured to: update the available node list using the information on whether the data can be received sent by other computing nodes.

[0133] As one of the implementable ways, the state determination unit 302 can be specifically configured to: record other computing nodes in the state of being able to receive data in the available node list by using the information sent by other computing nodes on whether the data can be received, and update the ranking of each computing node in the available node list by using the communication delay between other computing nodes. Accordingly, the forwarding processing unit 301 can be configured to: distribute the data to be processed to one or more computing nodes in the available node list according to the ranking results of each computing node in the available node list recorded by the current computing node.

[0134] As one of the achievable ways, the forwarding processing unit 301 may be specifically configured to: convert the data to be processed into a serialized format to obtain a byte stream; and transmit the byte stream to other computing nodes using a remote procedure call RPC.

[0135] Furthermore, the forwarding processing unit 301 may also be specifically configured to: receive the to-be-processed data from other computing nodes in the distributed system; and write the received to-be-processed data into the data queue corresponding to the current computing node.

[0136] As one of the achievable ways, the forwarding processing unit 301 may be specifically configured to: receive byte streams from other computing nodes in the distributed system; and process the byte streams using streaming RPC to obtain data to be processed.

[0137] like Figure 1 As shown in , an embodiment of the present application further provides a computing node, which includes a data distribution device, a data queue and a data processing device.

[0138] The data distribution device is configured to receive the data to be processed from the upstream system; determine whether the current computing node is in a state where it can receive data according to the state of the data queue corresponding to the current computing node; if the current computing node is in a state where it can receive data, write the data to be processed into the data queue corresponding to the current computing node; otherwise, forward the data to be processed to other computing nodes. The specific structure and functional principle of the data distribution device can be found in the Figure 4 Related notes of the illustrated embodiment.

[0139] The data processing device is configured to read the data to be processed from the data queue corresponding to the current computing node for processing.

[0140] Each embodiment in the current specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for system or device embodiments, since their basis is currently similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the current embodiment scheme. Ordinary technicians in the current field can understand and implement it without creative work.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0142] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0143] And an electronic device, comprising:

[0144] one or more processors; and

[0145] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0146] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0147] in, Figure 4 The architecture of the electronic device is shown as an example, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.

[0148] Among them, the processor 410 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0149] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operation of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and a data distribution device / computing node 425, etc. can also be stored. The above-mentioned data distribution device / computing node 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0150] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0151] The network interface 414 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0152] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420).

[0153] It should be noted that, although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0154] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0155] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A data distribution method, applied to a distributed system including a plurality of computing nodes, to which to-be-processed data from an upstream system are distributed, characterized in that: Each computing node executes as the current computing node: Receive data to be processed from upstream systems; Determining whether the current computing node is in a state capable of receiving data according to the state of the data queue corresponding to the current computing node; When the current computing node is in a state where it can receive data, the data to be processed is written into a data queue corresponding to the current computing node, so that the current computing node reads the data to be processed from the corresponding data queue for processing; When the current computing node is in a state where it cannot receive data, the data to be processed is forwarded to other computing nodes based on a list of available nodes recorded by the current node, wherein the list of available nodes includes computing nodes in the distributed system that are in a state where they can receive data except the current computing node.

2. The method according to claim 1, characterized in that Determining whether the current computing node is in a state capable of receiving data according to the state of the data queue corresponding to the current computing node comprises: In response to a state indicator value of a data queue corresponding to the current computing node being less than a first upper threshold, switching the current computing node to a state capable of receiving data, wherein the state indicator value is used to characterize the degree to which the data queue is occupied; In response to the state indicator value of the data queue corresponding to the current computing node being greater than a second upper limit threshold, switching the current computing node to a state where data cannot be received; The first upper threshold is less than or equal to the second upper threshold.

3. The method according to claim 2, characterized in that The method further comprises: When the actual resource occupancy status of the current computing node is lower than a first preset status for a certain period of time, increasing the first upper limit threshold and / or the second upper limit threshold; When the actual resource occupancy status of the current computing node is higher than a second preset status for a certain period of time, reducing the first upper limit threshold and / or the second upper limit threshold, and the first preset status is lower than or equal to the second preset status; The resource occupancy status includes at least one of a processor occupancy status, a memory occupancy status and a network bandwidth occupancy status.

4. The method according to claim 1, characterized in that: The method further comprises: The current computing node sends information about whether the current computing node is in a state capable of receiving data to other computing nodes in the distributed system; The available node list is updated by using the received information on whether the data can be received and sent by other computing nodes.

5. The method according to claim 4, characterized in that The updating of the available node list by using the received information on whether the data can be received sent by other computing nodes comprises: recording other computing nodes in the data receiving state in the available node list by using the information on whether the data can be received sent by other computing nodes, and updating the order of each computing node in the available node list by using the communication delay between other computing nodes; Forwarding the data to be processed to other computing nodes according to the available node list recorded by the current node includes: distributing the data to be processed to one or more computing nodes in the available node list according to the sorting results of each computing node in the available node list recorded by the current computing node.

6. The method according to claim 1, characterized in that Forwarding the to-be-processed data to other computing nodes includes: Convert the data to be processed into a serialized format to obtain a byte stream; The byte stream is transmitted to other computing nodes using remote procedure call (RPC).

7. The method according to claim 1, characterized in that The method further comprises: The current computing node receives data to be processed from other computing nodes in the distributed system; The received data to be processed is written into the data queue corresponding to the current computing node.

8. A data distribution device, applied to a distributed system including a plurality of computing nodes, to which the to-be-processed data from an upstream system are distributed, and the device is respectively arranged at each computing node; characterized in that: The device comprises: A forwarding processing unit, configured to receive data to be processed from an upstream system; A state determination unit, configured to determine whether the current computing node is in a state capable of receiving data according to a state of a data queue corresponding to the current computing node; The forwarding processing unit is further configured to, when the current computing node is in a state where it can receive data, write the data to be processed into a data queue corresponding to the current computing node, so that the current computing node can read the data to be processed from the corresponding data queue for processing; when the current computing node is in a state where it cannot receive data, forward the data to be processed to other computing nodes based on a list of available nodes recorded by the current node, wherein the list of available nodes includes computing nodes in the distributed system that are in a state where they can receive data except the current computing node.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images