Data shuffling flow control method and device and storage medium

By judging the current limiting requirements based on the delay time and request accumulation number of the target computing nodes in a distributed computing cluster, and automatically determining the application to be limited, the impact of data shuffling on cluster performance and stability is solved, and the performance of the cluster is realized and stability guarantee.

CN118921407APending Publication Date: 2024-11-08BEIJING VOLCANO ENGINE TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411119443.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In large-scale distributed computing clusters, the data shuffling process often becomes a bottleneck in performance and stability, resulting in abnormalities or large-scale data shuffling operations affecting the performance and stability of the distributed computing cluster.

Method used

By determining whether current limiting needs to be turned on based on the delay time and number of requested stacking of the target calculation node during the data shuffling process and the targeted limiting application, flow control is performed to alleviate the negative impact.

Benefits of technology

It effectively alleviates the negative impact of data shuffling on distributed computing clusters, reduces the risk of application failure, and ensures the performance and stability of distributed computing clusters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921407A_ABST
    Figure CN118921407A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data shuffling flow control method and device and a storage medium, and the method comprises the steps: judging whether a target computing node needs to start flow limiting or not according to the obtaining delay time and / or the request accumulation number of the target computing node in a distributed computing cluster in a data shuffling process; if it is determined that the target computing node needs to start flow limiting, determining a target application to be subjected to flow limiting from a plurality of applications requesting data from the target computing node in the data shuffling process; and performing current limiting on the target application according to the current request accumulation quantity of the target application. Whether flow limiting needs to be started or not is automatically judged in the data shuffling process of the target computing node in the distributed computing cluster, and the target application to be subjected to flow limiting is automatically determined, so that the negative influence of abnormal or large-scale data shuffling operation on the distributed computing cluster can be relieved; and application running failures caused by data shuffling data acquisition failures are reduced, so that the performance and the stability of the distributed computing cluster are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and more particularly to a data shuffling flow control method, device, and storage medium. Background Art

[0002] In distributed computing clusters such as Spark clusters, a data shuffle process is usually required. Data shuffle is the process of repartitioning and combining the data of all computing nodes.

[0003] However, in large-scale distributed computing clusters, data shuffling often becomes a bottleneck for performance and stability. The data shuffling process involves frequent disk and network read and write (IO) operations. The management unit on a single computing node, such as the ESS (External Shuffle Service) of each target computing node in the Spark cluster, needs to serve all data shuffle requests on the target node at the same time. Abnormal or large-scale data shuffling jobs may affect the performance and stability of the distributed computing cluster. Summary of the invention

[0004] The embodiments of the present disclosure provide a data shuffling flow control method, device, and storage medium to automatically perform traffic control on target computing nodes during the data shuffling process to ensure the performance and stability of a distributed computing cluster.

[0005] In a first aspect, an embodiment of the present disclosure provides a data shuffling flow control method, comprising:

[0006] According to the acquisition delay time and / or the number of request accumulation of the target computing node in the distributed computing cluster during the data shuffling process, it is determined whether the target computing node needs to enable current limiting;

[0007] If it is determined that the target computing node needs to enable current limiting, determining a target application to be current limited from multiple applications that request data from the target computing node during the data shuffling process;

[0008] According to the current number of requests accumulated by the target application, the target application is throttled.

[0009] In a second aspect, an embodiment of the present disclosure provides a data shuffling flow control device, including:

[0010] A judging unit, configured to judge whether the target computing node in the distributed computing cluster needs to start current limiting according to the acquisition delay time and / or the number of request accumulation of the target computing node in the data shuffling process;

[0011] A screening unit, configured to determine a target application to be limited in flow from a plurality of applications that request data from the target computing node during the data shuffling process if it is determined that the target computing node needs to enable flow limiting;

[0012] The execution unit is used to limit the flow of the target application according to the current request accumulation number of the target application.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data shuffling flow control method as described in the first aspect and various possible designs of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the data shuffling flow control method described in the first aspect and various possible designs of the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, it implements the data shuffling flow control method described in the first aspect and various possible designs of the first aspect.

[0018] The data shuffling flow control method, device and storage medium provided by the embodiments of the present disclosure judge whether the target computing node in the distributed computing cluster needs to turn on the flow limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the data shuffling process; if it is determined that the target computing node needs to turn on the flow limiting, the target application to be flow limited is determined from multiple applications that request data from the target computing node in the data shuffling process; and the target application is flow limited according to the current number of request accumulations of the target application. By automatically judging whether it is necessary to turn on the flow limiting during the data shuffling process of the target computing node in the distributed computing cluster, and automatically determining the target application to be flow limited, the negative impact of abnormal or large-scale data shuffling operations on the distributed computing cluster can be alleviated, and the application operation failure caused by the failure of data shuffling to obtain data can be reduced, thereby ensuring the performance and stability of the distributed computing cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of a scenario of a data shuffling flow control method provided by an embodiment of the present disclosure;

[0021] Figure 2 A flow chart of a method for controlling data shuffling flow provided by an embodiment of the present disclosure;

[0022] Figure 3 A flow chart of a data shuffling flow control method provided by another embodiment of the present disclosure;

[0023] Figure 4 A structural block diagram of a data shuffling flow control device provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0026] In large-scale distributed computing clusters, data shuffling often becomes a bottleneck for performance and stability. The data shuffling process involves frequent disk and network read and write (IO) operations. The management unit on a single computing node, such as the ESS of each target computing node in the Spark cluster, needs to serve all data shuffling requests on the target node at the same time. Abnormal or large-scale data shuffling jobs may affect the performance and stability of the distributed computing cluster.

[0027] In order to solve the above technical problems, the embodiment of the present disclosure provides a flow control method for data shuffling, by judging whether the target computing node needs to turn on flow limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process; if it is determined that the target computing node needs to turn on flow limiting, the target application to be flow limited is determined from multiple applications that request data from the target computing node during the data shuffling process; and the target application is flow limited according to the current number of request accumulations of the target application. By automatically judging whether it is necessary to turn on flow limiting and automatically determining the target application to be flow limited during the data shuffling process of the target computing node in the distributed computing cluster, the negative impact of abnormal or large-scale data shuffling operations on the distributed computing cluster can be alleviated, and the application operation failure caused by the failure of data shuffling to obtain data can be reduced, thereby ensuring the performance and stability of the distributed computing cluster.

[0028] The application scenarios of the data shuffling flow control method disclosed in the present invention are as follows: Figure 1 As shown, a distributed computing cluster includes multiple computing nodes. For any target computing node, data requests sent by applications in other computing nodes (such as Spark applications) can be received. Different applications can be deployed in different other computing nodes, or the same application (for example, distributed computing) can be deployed. Whether the target computing node needs to turn on current limiting can be determined based on the acquisition delay time and / or the number of request accumulations of the target computing node during the data shuffling process; if it is determined that the target computing node needs to turn on current limiting, the target application to be limited is determined from the multiple applications that request data from the target computing node during the data shuffling process; and the target application is limited based on the current number of request accumulations of the target application.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] The data shuffling flow control method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0031] refer to Figure 2 , Figure 2A flow chart of a method for controlling the flow of data shuffling provided in an embodiment of the present disclosure. The method of this embodiment can be applied to any target computing node in a distributed computing cluster, and its execution subject can be a processing unit for executing data shuffling in the target computing node, such as an ESS of each target computing node in a Spark cluster. The method for controlling the flow of data shuffling in this embodiment includes:

[0032] S201. Determine whether current limiting needs to be enabled for a target computing node in a distributed computing cluster according to the acquisition delay time and / or the number of request accumulations of the target computing node during a data shuffling process.

[0033] In this embodiment, for distributed computing clusters such as Spark clusters, it is usually necessary to perform a data shuffle process, which is a computing program that repartitions and combines the data of all computing nodes to redistribute the data and allow some data to converge on the same computing node for calculation. During the data shuffle process, the target computing node receives data requests (Shuffle requests) sent by other computing nodes, and returns the data to other computing nodes through read and write (IO) operations (specifically including Shuffle Read and Shuffle Write). In large-scale distributed computing clusters, the data shuffle process often becomes a bottleneck for performance and stability. In order to ensure the performance and stability of the distributed computing cluster, the flow control can be performed on each computing node of the distributed computing cluster. Specifically, it can be determined whether each computing node needs to turn on current limiting. If it is determined that current limiting needs to be turned on, current limiting is automatically performed.

[0034] In specific implementation, for any target computing node, two indicators can be obtained from the management unit of the target computing node (for example, the ESS of each target computing node in the Spark cluster): Fetch Latency and / or Queued Chunks. Fetch Latency refers to the time from the start of requesting data to obtaining data, reflecting the data transmission efficiency and performance of the target computing node; and Queued Chunks refers to the number of requests waiting to be processed in the management unit of the target computing node, reflecting the load situation and processing capacity.

[0035] Furthermore, whether the target computing node needs to enable current limiting can be determined based on the fetch delay time (Fetch Latency) and / or the size of the request accumulation number (Queued Chunks). The longer the fetch delay time (Fetch Latency) and the larger the number of request accumulation (Queued Chunks), the greater the need to enable current limiting on the target computing node.

[0036] In an optional embodiment, preset thresholds may be configured for the fetch delay time (Fetch Latency) and the number of request accumulations (Queued Chunks), respectively, and the fetch delay time (Fetch Latency) and / or the number of request accumulations (Queued Chunks) may be compared with the corresponding preset thresholds. If it is determined that the fetch delay time (Fetch Latency) and / or the number of request accumulations (Queued Chunks) exceed the corresponding preset thresholds, it is determined that the target computing node needs to enable current limiting, wherein at least one of the fetch delay time (Fetch Latency) and the number of request accumulations (Queued Chunks) exceeds the corresponding preset thresholds, then it is determined that the target computing node needs to enable current limiting; of course, it may also be that the fetch delay time (Fetch Latency) and the number of request accumulations (Queued Chunks) need to exceed the corresponding preset thresholds at the same time before it is determined that the target computing node needs to enable current limiting.

[0037] Of course, other methods can also be used to determine whether the target computing node needs to enable current limiting based on the size of the fetch latency and / or the number of queued chunks. For example, an evaluation index is determined based on the fetch latency and / or the number of queued chunks, and whether the target computing node needs to enable current limiting is determined based on the evaluation index, etc. Examples are not given one by one here.

[0038] S202: If it is determined that the target computing node needs to enable current limiting, determine a target application to be current limited from multiple applications that request data from the target computing node during the data shuffling process.

[0039] In this embodiment, when it is determined that the target computing node needs to turn on current limiting, the current limiting of the target computing node is turned on. However, during the data shuffling process, there may be multiple applications (such as Spark applications, distributed in the computing nodes) that request data from the target computing node. Different applications send different numbers of requests to the target computing node per unit time and have different priorities. Therefore, current limiting may not be performed on all applications. In this embodiment, one or more target applications to be limited can be screened out from multiple applications, where the target application can be an application with a large number of requests per unit time (such as QPS), an application with a lower priority, or an application that meets other conditions.

[0040] Optionally, in this embodiment, a preset unit time request threshold can be configured for the multiple applications. However, considering that the priority of each application is different, a corresponding preset unit time request threshold can be configured for each application respectively. If the current unit time request number of any application among the multiple applications exceeds the preset unit time request number threshold corresponding to the application, the application will be determined as the target application to be limited, thereby ensuring that high-priority applications can stably request data and thus ensure their stable operation.

[0041] Optionally, the preset request quantity threshold per unit time corresponding to any application may be related to the maximum number of received requests per unit time allowed by the target computing node, the number of applications requesting data from the target computing node, and the priority of the application, wherein the maximum number of received requests per unit time allowed by the target computing node may be the maximum number of received requests per unit time of the target computing node in the previous time period. Therefore, the preset request quantity threshold per unit time corresponding to the application may be determined based on the maximum number of received requests per unit time of the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of the application.

[0042] Furthermore, in specific implementation, the maximum number of requests received per unit time of the target computing node in the previous time period and the number of applications currently requesting data from the target computing node can be used as the basis, and according to the priority of each application, the corresponding number of requests received per unit time can be allocated as the preset number of requests per unit time threshold corresponding to the application, that is, the maximum number of requests received per unit time of the target computing node in the previous time period is allocated based on the number of applications currently requesting data from the target computing node and the priority of the application, and a number of requests received per unit time is allocated to each application, which is determined as the preset number of requests per unit time threshold corresponding to each application. Since different applications have different priorities, the number of requests received per unit time allocated is also different. The higher the priority, the more requests received per unit time are allocated, and the higher the preset number of requests per unit time threshold is, that is, the preset number of requests per unit time threshold of applications with high priority is greater than the preset number of requests per unit time threshold of applications with low priority.

[0043] In an optional embodiment, the preset request quantity threshold per unit time corresponding to any application a can be calculated based on the following formula:

[0044]

[0045] Among them, T is the current time, ΔT is the duration of the previous time period, represents the maximum number of requests received per unit time by the target computing node in the previous time period; N is the number of applications currently requesting data from the target computing node; p is the priority of application a, and the lower the value, the higher the priority. The allocation is made based on N and p.

[0046] Furthermore, if the current number of requests per unit time for application a exceeds Then the application a is determined as the target application to be limited.

[0047] S203: Limit the flow of the target application according to the current number of requests accumulated by the target application.

[0048] In this embodiment, when the target application is subjected to current limiting, the current limiting may be performed based on the current number of requests accumulated by the target application, that is, limiting the speed or number of requests sent by the target application to the target computing node.

[0049] Optionally, a preset request backlog number threshold can be configured for each target application to be limited. However, taking into account the different situations of different applications, a preset request backlog number threshold can be configured for each target application to be limited respectively, and then the current request backlog number of any target application is compared with its corresponding preset request backlog number threshold. If the current request backlog number of the target application exceeds its corresponding preset request backlog number threshold, a stop request instruction can be sent to the target application to cause the target application to suspend sending requests to the target computing node, and / or refuse to receive requests sent by the target application (that is, even if the target application continues to send requests to the target computing node, it refuses to receive requests sent by the target application).

[0050] Optionally, when determining the preset request accumulation number threshold corresponding to any target application, the maximum number of requests per unit time in the previous time period may be considered, which can adapt to changes in the number of requests of the target application and avoid limiting the flow based on the unchanged preset request accumulation number threshold. Specifically, the preset request accumulation number threshold corresponding to the target application may be determined based on the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node.

[0051] In specific implementation, the product of the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node can be obtained, and the product is determined as the preset request accumulation number threshold corresponding to the target application. The formula is as follows:

[0052]

[0053] Among them, fra Indicates the number of requests per unit time of target application a at any time; Indicates the maximum number of requests per unit time of target application a in the previous time period; L θ Indicates the allowed accumulation time of the target computing node.

[0054] Furthermore, if the current number of requests accumulated by the target application a exceeds the corresponding preset request accumulation number threshold The target application a may be current limited, that is, a stop request instruction may be sent to the target application a to make the target application suspend sending requests to the target computing node and / or refuse to receive requests sent by the target application a.

[0055] The data shuffling flow control method provided in this embodiment determines whether the target computing node in the distributed computing cluster needs to turn on flow limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the data shuffling process; if it is determined that the target computing node needs to turn on flow limiting, the target application to be flow limited is determined from multiple applications that request data from the target computing node in the data shuffling process; and the target application is flow limited according to the current number of request accumulations of the target application. By automatically determining whether it is necessary to turn on flow limiting during the data shuffling process of the target computing node in the distributed computing cluster, and automatically determining the target application to be flow limited, the negative impact of abnormal or large-scale data shuffling operations on the distributed computing cluster can be alleviated, and application operation failures caused by failures in data shuffling to obtain data can be reduced, thereby ensuring the performance and stability of the distributed computing cluster.

[0056] Based on any of the above embodiments, after determining that the target computing node needs to turn on current limiting (or after turning on current limiting for the target settlement node), if it is determined that the target computing node does not need current limiting based on the acquisition delay time and / or the number of request accumulations during the data shuffling process of the target computing node, then the current limiting can be stopped for the target computing node.

[0057] Optionally, in order to avoid the target computing node frequently starting and stopping current limiting due to fluctuations in the acquisition delay time and / or the number of request accumulations, thereby causing instability, in this embodiment, after determining that the target computing node needs to start current limiting (or after starting current limiting for the target settlement node), specifically as follows Figure 3 As shown, it may include:

[0058] S301: If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then it is determined whether the throttling duration of the target computing node exceeds the preset throttling duration;

[0059] S302: If the current limiting duration of the target computing node exceeds the preset current limiting duration, stop current limiting for the target computing node; or

[0060] S303: If the current limiting duration of the target computing node does not exceed the preset current limiting duration, continue to enable current limiting until the current limiting duration exceeds the preset current limiting duration, and stop current limiting for the target computing node.

[0061] That is, in this embodiment, after the current limiting is turned on, the current limiting is maintained for at least a preset current limiting time period (for example, half an hour), and fluctuations within the preset current limiting time period can be avoided.

[0062] In addition, after limiting the flow of any target application, if it is determined that the target application does not need to be limited, for example, the current unit time request number of the target application no longer exceeds the preset unit time request number threshold corresponding to the application, the target application can be stopped from being limited.

[0063] Furthermore, also taking into account the fluctuation problem, the current limiting can be maintained for at least the preset current limiting duration after the target application starts the current limiting. That is, if it is determined that the target application does not need current limiting, it is determined whether the current limiting duration of the target application exceeds the preset current limiting duration; if the current limiting duration of the target application exceeds the preset current limiting duration, the current limiting for the target application is stopped; or, if the current limiting duration of the target application does not exceed the preset current limiting duration, the current limiting for the target application continues until the current limiting duration exceeds the preset current limiting duration, and the current limiting for the target application is stopped.

[0064] Corresponding to the data shuffling flow control method of the above embodiment, Figure 4 The structural block diagram of the data shuffling flow control device provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 4 The data shuffling flow control device 400 includes: a judgment unit 401, a screening unit 402, and an execution unit 403.

[0065] The judging unit 401 is used to judge whether the target computing node needs to start current limiting according to the acquisition delay time and / or the number of request accumulation of the target computing node in the distributed computing cluster during the data shuffling process;

[0066] A screening unit 402 is configured to determine a target application to be limited from multiple applications that request data from the target computing node during the data shuffling process if it is determined that the target computing node needs to enable current limiting;

[0067] The execution unit 403 is used to limit the flow of the target application according to the current request accumulation number of the target application.

[0068] In one or more embodiments of the present disclosure, when the determination unit 401 determines whether the target computing node needs to start current limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, the determination unit 401 is used to:

[0069] Obtaining a preset threshold value corresponding to the acquisition delay time and / or the request accumulation quantity;

[0070] If it is determined that the acquisition delay time and / or the number of request accumulations exceeds corresponding preset thresholds, it is determined that the target computing node needs to enable current limiting.

[0071] In one or more embodiments of the present disclosure, when determining a target application to be limited from among multiple applications requesting data from the target computing node during the data shuffling process, the screening unit 402 is configured to:

[0072] Obtaining the current number of requests per unit time of each application requesting data from the target computing node;

[0073] Determine a preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node;

[0074] If the current number of requests per unit time of any application among the multiple applications exceeds a preset number of requests per unit time threshold corresponding to the any application, the any application is determined as a target application to be limited.

[0075] In one or more embodiments of the present disclosure, when determining the preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node, the screening unit 402 is used to:

[0076] The preset request quantity threshold per unit time corresponding to any application is determined based on the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application.

[0077] In one or more embodiments of the present disclosure, when determining the preset request quantity threshold per unit time corresponding to any application according to the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application, the screening unit 402 is used to:

[0078] Based on the maximum number of requests received per unit time of the target computing node in the previous time period and the number of applications currently requesting data from the target computing node, a corresponding number of requests received per unit time is allocated to each application according to its priority as the preset unit time request number threshold corresponding to the application, wherein the preset unit time request number threshold of a high-priority application is greater than the preset unit time request number threshold of a low-priority application.

[0079] In one or more embodiments of the present disclosure, when limiting the flow of the target application according to the current number of requests accumulated by the target application, the execution unit 403 is used to:

[0080] If the current request accumulation number of the target application exceeds the preset request accumulation number threshold corresponding to the target application, a stop request instruction is sent to the target application and / or the request sent by the target application is refused.

[0081] In one or more embodiments of the present disclosure, the execution unit 403 is further configured to:

[0082] A preset request accumulation quantity threshold corresponding to the target application is determined according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node.

[0083] In one or more embodiments of the present disclosure, when determining the preset request accumulation quantity threshold corresponding to the target application according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation duration allowed by the target computing node, the execution unit 403 is used to:

[0084] The product of the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node is obtained, and the product is determined as a preset request accumulation number threshold corresponding to the target application.

[0085] In one or more embodiments of the present disclosure, after determining that the target computing node needs to enable current limiting, the execution unit 403 is further configured to:

[0086] If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, the throttling of the target computing node is stopped.

[0087] In one or more embodiments of the present disclosure, if the execution unit 403 determines that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then when stopping the throttling of the target computing node, it is used to:

[0088] If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then it is determined whether the throttling duration of the target computing node exceeds the preset throttling duration;

[0089] If the current limiting duration of the target computing node exceeds the preset current limiting duration, stopping the current limiting of the target computing node; or

[0090] If the current limiting duration of the target computing node does not exceed the preset current limiting duration, the current limiting continues until the current limiting duration exceeds the preset current limiting duration, and the current limiting of the target computing node is stopped.

[0091] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0092] refer to Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure, and the electronic device 500 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0093] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0094] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0096] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0097] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0098] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0099] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0100] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0101] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0102] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0103] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] In a first aspect, according to one or more embodiments of the present disclosure, a data shuffling flow control method is provided, comprising:

[0105] According to the acquisition delay time and / or the number of request accumulation of the target computing node in the distributed computing cluster during the data shuffling process, it is determined whether the target computing node needs to enable current limiting;

[0106] If it is determined that the target computing node needs to enable current limiting, determining a target application to be current limited from multiple applications that request data from the target computing node during the data shuffling process;

[0107] According to the current number of requests accumulated by the target application, the target application is throttled.

[0108] According to one or more embodiments of the present disclosure, judging whether the target computing node needs to enable current limiting according to the acquisition delay time and / or the number of request accumulation of the target computing node in the distributed computing cluster during the data shuffling process includes:

[0109] Obtaining a preset threshold value corresponding to the acquisition delay time and / or the request accumulation quantity;

[0110] If it is determined that the acquisition delay time and / or the number of request accumulations exceeds corresponding preset thresholds, it is determined that the target computing node needs to enable current limiting.

[0111] According to one or more embodiments of the present disclosure, determining a target application to be limited from multiple applications requesting data from the target computing node during the data shuffling process includes:

[0112] Obtaining the current number of requests per unit time of each application requesting data from the target computing node;

[0113] Determine a preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node;

[0114] If the current number of requests per unit time of any application among the multiple applications exceeds a preset number of requests per unit time threshold corresponding to the any application, the any application is determined as a target application to be limited.

[0115] According to one or more embodiments of the present disclosure, determining a preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node includes:

[0116] The preset request quantity threshold per unit time corresponding to any application is determined based on the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application.

[0117] According to one or more embodiments of the present disclosure, determining a preset request quantity threshold per unit time corresponding to any application according to the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application, includes:

[0118] Based on the maximum number of requests received per unit time of the target computing node in the previous time period and the number of applications currently requesting data from the target computing node, a corresponding number of requests received per unit time is allocated to each application according to its priority as the preset unit time request number threshold corresponding to the application, wherein the preset unit time request number threshold of a high-priority application is greater than the preset unit time request number threshold of a low-priority application.

[0119] According to one or more embodiments of the present disclosure, limiting the flow of the target application according to the current number of requests accumulated by the target application includes:

[0120] If the current request accumulation number of the target application exceeds the preset request accumulation number threshold corresponding to the target application, a stop request instruction is sent to the target application and / or the request sent by the target application is refused.

[0121] According to one or more embodiments of the present disclosure, the method further includes:

[0122] A preset request accumulation quantity threshold corresponding to the target application is determined according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node.

[0123] According to one or more embodiments of the present disclosure, determining a preset request accumulation number threshold corresponding to the target application according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node includes:

[0124] The product of the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node is obtained, and the product is determined as a preset request accumulation number threshold corresponding to the target application.

[0125] According to one or more embodiments of the present disclosure, after determining that the target computing node needs to enable current limiting, the method further includes:

[0126] If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, the throttling of the target computing node is stopped.

[0127] According to one or more embodiments of the present disclosure, if it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the data shuffling process in the distributed computing cluster, then stopping the throttling of the target computing node includes:

[0128] If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then it is determined whether the throttling duration of the target computing node exceeds the preset throttling duration;

[0129] If the current limiting duration of the target computing node exceeds the preset current limiting duration, stopping the current limiting of the target computing node; or

[0130] If the current limiting duration of the target computing node does not exceed the preset current limiting duration, the current limiting continues until the current limiting duration exceeds the preset current limiting duration, and the current limiting of the target computing node is stopped.

[0131] In a second aspect, according to one or more embodiments of the present disclosure, a data shuffling flow control device is provided, comprising:

[0132] A judging unit, configured to judge whether the target computing node in the distributed computing cluster needs to start current limiting according to the acquisition delay time and / or the number of request accumulation of the target computing node in the data shuffling process;

[0133] A screening unit, configured to determine a target application to be limited in flow from a plurality of applications that request data from the target computing node during the data shuffling process if it is determined that the target computing node needs to enable flow limiting;

[0134] The execution unit is used to limit the flow of the target application according to the current request accumulation number of the target application.

[0135] According to one or more embodiments of the present disclosure, when the determination unit determines whether the target computing node needs to start current limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, the determination unit is used to:

[0136] Obtaining a preset threshold value corresponding to the acquisition delay time and / or the request accumulation quantity;

[0137] If it is determined that the acquisition delay time and / or the number of request accumulations exceeds corresponding preset thresholds, it is determined that the target computing node needs to enable current limiting.

[0138] According to one or more embodiments of the present disclosure, when the screening unit determines the target application to be limited from among the multiple applications requesting data from the target computing node during the data shuffling process, it is configured to:

[0139] Obtaining the current number of requests per unit time of each application requesting data from the target computing node;

[0140] Determine a preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node;

[0141] If the current number of requests per unit time of any application among the multiple applications exceeds a preset number of requests per unit time threshold corresponding to the any application, the any application is determined as a target application to be limited.

[0142] According to one or more embodiments of the present disclosure, when determining the preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node, the screening unit is used to:

[0143] The preset request quantity threshold per unit time corresponding to any application is determined based on the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application.

[0144] According to one or more embodiments of the present disclosure, when determining the preset request quantity threshold per unit time corresponding to any application according to the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application, the screening unit is used to:

[0145] Based on the maximum number of requests received per unit time of the target computing node in the previous time period and the number of applications currently requesting data from the target computing node, a corresponding number of requests received per unit time is allocated to each application according to its priority as the preset unit time request number threshold corresponding to the application, wherein the preset unit time request number threshold of a high-priority application is greater than the preset unit time request number threshold of a low-priority application.

[0146] According to one or more embodiments of the present disclosure, when the execution unit performs flow limiting on the target application according to the current request accumulation number of the target application, it is configured to:

[0147] If the current request accumulation number of the target application exceeds the preset request accumulation number threshold corresponding to the target application, a stop request instruction is sent to the target application and / or the request sent by the target application is refused.

[0148] According to one or more embodiments of the present disclosure, the execution unit is further configured to:

[0149] A preset request accumulation quantity threshold corresponding to the target application is determined according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node.

[0150] According to one or more embodiments of the present disclosure, when determining the preset request accumulation number threshold corresponding to the target application according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node, the execution unit is used to:

[0151] The product of the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node is obtained, and the product is determined as a preset request accumulation number threshold corresponding to the target application.

[0152] According to one or more embodiments of the present disclosure, after determining that the target computing node needs to enable current limiting, the execution unit is further configured to:

[0153] If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, the throttling of the target computing node is stopped.

[0154] According to one or more embodiments of the present disclosure, if the execution unit determines that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then when stopping the throttling of the target computing node, it is used to:

[0155] If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then it is determined whether the throttling duration of the target computing node exceeds the preset throttling duration;

[0156] If the current limiting duration of the target computing node exceeds the preset current limiting duration, stopping the current limiting of the target computing node; or

[0157] If the current limiting duration of the target computing node does not exceed the preset current limiting duration, the current limiting continues until the current limiting duration exceeds the preset current limiting duration, and the current limiting of the target computing node is stopped.

[0158] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0159] The memory stores computer-executable instructions;

[0160] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data shuffling flow control method as described in the first aspect and various possible designs of the first aspect.

[0161] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data shuffling flow control method described in the first aspect and various possible designs of the first aspect is implemented.

[0162] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the data shuffling flow control method as described in the first aspect and various possible designs of the first aspect is implemented.

[0163] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0164] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0165] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data shuffling flow control method, characterized in that: include: According to the acquisition delay time and / or the number of request accumulation of the target computing node in the distributed computing cluster during the data shuffling process, it is determined whether the target computing node needs to enable current limiting; If it is determined that the target computing node needs to enable current limiting, determining a target application to be current limited from multiple applications that request data from the target computing node during the data shuffling process; According to the current number of requests accumulated by the target application, the target application is throttled.

2. The method according to claim 1, characterized in that The step of judging whether the target computing node needs to start current limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process includes: Obtaining a preset threshold value corresponding to the acquisition delay time and / or the request accumulation quantity; If it is determined that the acquisition delay time and / or the number of request accumulations exceeds corresponding preset thresholds, it is determined that the target computing node needs to enable current limiting.

3. The method according to claim 1, characterized in that Determining a target application to be limited from multiple applications requesting data from the target computing node during the data shuffling process includes: Obtaining the current number of requests per unit time of each application requesting data from the target computing node; Determine a preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node; If the current number of requests per unit time of any application among the multiple applications exceeds a preset number of requests per unit time threshold corresponding to the any application, the any application is determined as a target application to be limited.

4. The method according to claim 3, characterized in that: The determining of a preset request quantity threshold per unit time corresponding to each application requesting data from the target computing node includes: The preset request quantity threshold per unit time corresponding to any application is determined based on the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any application.

5. The method according to claim 4, characterized in that The determining, according to the maximum number of requests received per unit time by the target computing node in the previous time period, the number of applications currently requesting data from the target computing node, and the priority of any one of the applications, a preset threshold value of the number of requests per unit time corresponding to any one of the applications, includes: Based on the maximum number of requests received per unit time of the target computing node in the previous time period and the number of applications currently requesting data from the target computing node, a corresponding number of requests received per unit time is allocated to each application according to its priority as the preset unit time request number threshold corresponding to the application, wherein the preset unit time request number threshold of a high-priority application is greater than the preset unit time request number threshold of a low-priority application.

6. The method according to claim 1, characterized in that The limiting the flow of the target application according to the current number of requests accumulated by the target application includes: If the current request accumulation number of the target application exceeds the preset request accumulation number threshold corresponding to the target application, a stop request instruction is sent to the target application and / or the request sent by the target application is refused.

7. The method according to claim 6, characterized in that The method further comprises: A preset request accumulation quantity threshold corresponding to the target application is determined according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node.

8. The method according to claim 7, characterized in that The determining, according to the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node, a preset request accumulation number threshold corresponding to the target application includes: The product of the maximum number of requests per unit time of the target application in the previous time period and the accumulation time allowed by the target computing node is obtained, and the product is determined as a preset request accumulation number threshold corresponding to the target application.

9. The method according to claim 1, characterized in that: After determining that the target computing node needs to enable current limiting, the method further includes: If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, the throttling of the target computing node is stopped.

10. The method according to claim 9, characterized in that If it is determined that the target computing node does not need current limiting according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then current limiting of the target computing node is stopped, including: If it is determined that the target computing node does not need to be throttled according to the acquisition delay time and / or the number of request accumulations of the target computing node in the distributed computing cluster during the data shuffling process, then it is determined whether the throttling duration of the target computing node exceeds the preset throttling duration; If the current limiting duration of the target computing node exceeds the preset current limiting duration, stopping the current limiting of the target computing node; or If the current limiting duration of the target computing node does not exceed the preset current limiting duration, the current limiting continues until the current limiting duration exceeds the preset current limiting duration, and the current limiting of the target computing node is stopped.

11. A data shuffling flow control device, characterized in that: include: A judging unit, configured to judge whether the target computing node in the distributed computing cluster needs to start current limiting according to the acquisition delay time and / or the number of request accumulation of the target computing node in the data shuffling process; A screening unit, configured to determine a target application to be limited in flow from a plurality of applications that request data from the target computing node during the data shuffling process if it is determined that the target computing node needs to enable flow limiting; The execution unit is used to limit the flow of the target application according to the current request accumulation number of the target application.

12. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 10 is implemented.

Citation Information

Cited By

  • Spark operation flow control method, system and device

    CN119906637A